feat: add native CUDA Sol-Attn support

docs/backend.md

@@ -156,11 +156,8 @@ the runner's graph-cut capacity checks.
 
 Runtime capacity checks also leave 512 MiB of currently free device memory for
 backend scratch buffers and pipelines, including with explicit backend assignments.
-They cap free-memory reports by the device's total memory minus tracked
-resident allocations. Vulkan reports exceeding total memory are rejected because
-its heap-budget subtraction can underflow. Other backends use the cap instead of
-treating such reports as zero free memory. Failed checks log the reported free and
-total memory alongside tracked weight and runtime allocations.
+They cap stale free-memory reports by the device's total memory minus tracked
+resident allocations and reject reports that exceed the device's total memory.
 
 Components are considered in `diffusion`, `te`, `vae` order so that repeatedly
 used diffusion weights have priority. Each component's weights use the first

docs/edit.md

@@ -26,9 +26,6 @@ Stable-diffusion.spp also supports basic Unet-based editing models like instruct
 
 ## Configuring Reference Modes (`--ref-image-args`)
 
-For a one-time input transform before reference presets and model processing,
-including cropping, padding, and resizing algorithms, see [Image preprocessing](./image_preprocessing.md).
-
 Different DiT-based editing models require different configurations to process reference images correctly (e.g., whether to use a Vision Language Model (VLM) encoder or pass VAE-encoded images directly to the DiT).
 
 To simplify this, we provide **Presets**. By default, the system automatically selects the best preset based on the model architecture. However, you can override this using the `--ref-image-args` argument.

docs/esrgan.md

@@ -2,8 +2,6 @@
 
 You can use ESRGAN—such as the model [RealESRGAN_x4plus_anime_6B.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)—to upscale the generated images and improve their overall resolution and clarity.
 
-RGBA images, including Qwen Image 2.1 output, keep their alpha channel during model upscaling and hires fix. ESRGAN processes the RGB channels; the alpha channel is resized with bilinear interpolation and recombined with the upscaled image.
-
 - Specify the model path using the `--upscale-model PATH` parameter. example:
 
 ```bash

docs/image_preprocessing.md

@@ -1,173 +0,0 @@
-# Image preprocessing
-
-Use `--image-preprocess` to transform each image input once, before generation:
-
-```sh
-sd-cli ... \
-  --image-preprocess "target=init,mode=crop-resize,filter=lanczos,antialias=true" \
-  --image-preprocess "target=mask,filter=nearest-exact" \
-  --image-preprocess "target=ref,index=0,mode=fit-pad,width=768,height=768,filter=bicubic"
-```
-
-CLI and server image loaders decode at the original resolution. The generation
-entry point merges input defaults with user rules and prepares one transformed
-image per input. The original pipeline then consumes those images, including
-its mandatory canvas adaptation, reference resizing, and encoder preprocessing.
-
-```text
-native-resolution image
-  -> input defaults + user overrides
-  -> one input transform
-  -> original generation pipeline and model-specific processing
-```
-
-These rules do not override internal VAE, CLIP/VLM, ControlNet, or pixel-patch preprocessing.
-`--ref-image-args` retains its existing meaning and runs after this input transform.
-
-## Inputs and defaults
-
-| `target` | Input | Default geometry | Indexed? |
-| --- | --- | --- | --- |
-| `init` | img2img image or video first frame | Center crop to the generation aspect ratio, then resize | No |
-| `end` | Video last frame | Center crop, then resize | No |
-| `mask` | Inpainting mask | Inherit init geometry; otherwise center crop, then resize | No |
-| `control` | Control image | Center crop, then resize | No |
-| `ref` | Reference images | Preserve source dimensions | Yes |
-| `ip-adapter` | IP-Adapter image | Preserve source dimensions | No |
-| `id` | PhotoMaker identity images | Preserve source dimensions | Yes |
-| `control-frame` | Control video frames | Center crop, then resize | Yes |
-
-Canvas defaults use the aligned generation dimensions. Reference, IP-Adapter,
-and identity inputs use their original dimensions unless overridden. Default
-resampling is nearest for images and nearest-exact for masks.
-
-These defaults are shared by CLI, server, and C API. Moving geometry out of
-the loaders replaces the previous CLI/server BOX/sRGB resizing, so default
-pixels are not guaranteed to match earlier builds.
-
-Reference video and audio preprocessing are outside these image rules.
-Preprocessing options apply to `img_gen` and `vid_gen`, not standalone upscale
-or ADetailer mode. ADetailer clears the user's rules for its internal crops.
-
-## Rules
-
-Rules are comma-separated `key=value` lists. Repeat the CLI option or separate
-rules with semicolons. Every rule requires a `target` and at least one option.
-Rule syntax and input compatibility are checked when image/video generation
-starts. Unknown keys, invalid values, duplicate keys in a rule, missing images,
-and out-of-range indices cause generation to fail with an error log.
-
-Omit `index` to configure every image of that type; otherwise use a zero-based
-index. CLI directory inputs follow filename order. Indexed rules override
-type-wide rules field by field, regardless of order. At equal specificity,
-the last value for a field wins. `auto` selects the input preset.
-
-| `mode` | Input transform |
-| --- | --- |
-| `auto` | Use the input's default geometry |
-| `none` | Keep source dimensions without resizing, cropping, or padding |
-| `stretch` | Resize to the target dimensions |
-| `crop` | Crop a target-sized rectangle without resizing; fail if the source is too small |
-| `crop-resize` | Crop to the target aspect ratio, then resize |
-| `fit-pad` | Fit the entire image inside the target dimensions, preserving aspect ratio, then pad |
-
-`width` and `height` must be specified together as positive integers. They
-override the input transform's dimensions, not the generation or encoder size.
-For a native-size preset, specifying dimensions without a mode selects stretch.
-`mode=none` with explicit dimensions different from the source is contradictory
-and is rejected.
-
-`anchor=center|top|bottom|left|right` selects crop/padding placement.
-`pad_color=#RRGGBB` or `#RRGGBBAA` selects padding, defaulting to opaque black.
-A grayscale mask uses the first color component.
-
-`filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos` selects resampling.
-`antialias=auto|true|false` enables antialiasing automatically for filtered
-downscaling; explicit true requires bilinear, bicubic, or Lanczos.
-Filtered RGBA resizing uses premultiplied alpha.
-
-`canny=true|false` enables edge detection for any supported image target,
-defaulting to `false`. It runs once after geometry, before the original
-generation pipeline, including with `mode=none`. Grayscale, grayscale-alpha,
-RGB, and RGBA inputs are supported; alpha is preserved.
-
-Each input has its own Canny setting. Indexed rules can enable or disable it
-for individual references, identity images, or video control frames.
-
-```sh
---image-preprocess "target=init,mode=fit-pad,canny=true"
---image-preprocess "target=ref,index=0,mode=none,canny=true"
---image-preprocess "target=control-frame,index=2,canny=true"
-```
-
-Init and mask sources must have the same dimensions. The mask inherits the
-init crop, resize, and padding coordinates, while retaining its own filter,
-padding value, and Canny setting. Conflicting mask geometry is rejected. An
-omitted mask remains absent until the original pipeline creates its default mask.
-
-## Downstream behavior
-
-`mode=none` only skips the input geometry transform. For example:
-
-```sh
---image-preprocess "target=init,mode=none" \
---image-preprocess "target=ref,mode=none"
-```
-
-The init image is still adapted to the generation canvas by the original
-pipeline. Reference images still follow `--ref-image-args` and model-specific
-resizing. CLIP retains its fixed input dimensions and normalization. HiDream-O1
-retains its original pixel-reference and visual preprocessing.
-
-Existing sharing between consumers is preserved: for example, Wan img2video
-uses the same adapted first frame for VAE conditioning and CLIP. High-resolution
-passes reuse the prepared images and apply their original size adaptation;
-they do not apply the user's crop a second time.
-
-To disable reference resizing before VAE encoding, use
-`--ref-image-args "resize_before_vae=false"` or the server field
-`"ref_image_args": "resize_before_vae=false"`. This is separate from
-`target=ref,mode=none`, which only skips input geometry. Model constraints
-still apply.
-
-## Server requests
-
-Native image/video requests and SDAPI accept `image_preprocess` as a string or
-an array of rule strings:
-
-```json
-{
-  "image_preprocess": [
-    "target=init,mode=fit-pad,filter=bicubic",
-    "target=mask,filter=nearest-exact",
-    "target=ref,index=0,mode=none"
-  ]
-}
-```
-
-OpenAI-compatible requests accept it through
-`<sd_cpp_extra_args>{...}</sd_cpp_extra_args>` in the prompt.
-Request rules replace server-default rules. Generation metadata records the
-user rules; image encodings and channel conventions are unchanged.
-
-## C API
-
-Set `image_preprocess` on the existing image/video generation parameters.
-The `generate_image()` and `generate_video()` signatures are unchanged:
-
-```c
-sd_img_gen_params_t params;
-sd_img_gen_params_init(&params);
-/* Set prompt, original-resolution input images, and generation options. */
-params.image_preprocess.rules = "target=init,mode=crop-resize,filter=lanczos;"
-                                "target=mask,filter=nearest-exact";
-bool ok = generate_image(ctx, &params, &images, &count);
-```
-
-Both generation parameter initializers set `image_preprocess.rules` to `NULL`,
-selecting input presets. Rule strings are borrowed for the synchronous call.
-The library owns temporary transformed pixels; caller images and arrays are
-not modified. Add `canny=true` to the desired target's rule in
-`image_preprocess.rules` to enable Canny.
-
-The parameter structs have grown; applications and bindings must be rebuilt.

docs/qwen_image_2.1.md

@@ -39,21 +39,3 @@ Pass the reference image with `-r` and describe the edit in `-p`. Vision weights
 ```
 
 For multiple reference images, repeat `-r` in the desired order, for example `-r first.png -r second.png`.
-
-### Alpha channel
-
-This model supports alpha channel output. As the model determines whether to output a regular image or with transparency through the prompt, according to [official recommendation](https://github.com/QwenLM/Qwen-Image-2.1#transparent-image-generation-rgba), use the following prompt format for better results:
-
-> `This is an RGBA image with transparency. <your description>. The image has alpha channel and the background is transparent.`
-
-Since transparency is decided by the prompt rather than by the input or an explicit switch, the same format applies equally to editing, whether or not the reference image itself has an alpha channel. Note that alpha is kept only in `.png` and `.webp` outputs; saving as `.jpg` drops the transparency.
-
-Here are some examples ran with Q6_K quantization:
-| Input | Prompt | Output |
-| --- | --- | --- |
-| ![Qwen Image 2.1 alpha input example 1](../assets/qwen/qwen-image-2.1-alpha-in1.png) | This is an RGBA image with transparency. Replace the text "BLOOM" with "Qwen Image 2.1", keeping the same font of the original text. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 1](../assets/qwen/qwen-image-2.1-alpha-out1.png) |
-| ![Qwen Image 2.1 alpha input example 2](../assets/logo.png) | This is an RGBA image with transparency. Remove the background of the image, keeping only the text and cat. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 2](../assets/qwen/qwen-image-2.1-alpha-out2.png) |
-
-### Other features
-
-Other features of the model could be found on the [model card from QwenLM/Qwen-Image-2.1 repo](https://github.com/QwenLM/Qwen-Image-2.1), including 2 finetuned prompt rewriting Qwen3.5-9B model.

docs/sol_attention.md

@@ -0,0 +1,106 @@
+# Sol-Attn
+
+`--sol-attn` enables native CUDA Sol-Attn in the diffusion model, including the
+high-noise diffusion model when present. It uses the shared attention dispatcher
+without classifying tokens as text, images, or video. Python, PyTorch, Triton,
+and CuTe DSL are not needed to build or run it.
+
+This implementation follows the diagonal-threshold algorithm in
+[NVlabs/Sana's Sol-Attn](https://github.com/NVlabs/Sana/tree/sol-engine/techniques/sparse_backends/sol_attn).
+It summarizes 64-token KV blocks, selects exact blocks using proxy scores and
+an online threshold, and approximates the remaining blocks using their K means
+and V sums. Adjacent blocks remain exact. Both contributions share an online
+softmax normalizer. Q/K/V and probability tiles use BF16 Tensor Cores with FP32
+accumulation; the BF16 result is returned through the existing FP32 interface.
+
+## Build
+
+Use patched GGML, CUDA Toolkit 12.0 or newer, and an NVIDIA GPU with compute
+capability 8.0 or newer. Compile kernels for the target GPU:
+
+```sh
+cmake -S . -B build -DSD_CUDA=ON -DSD_USE_UPSTREAM_GGML=OFF
+cmake --build build --config Release
+```
+
+The feature is compiled with the CUDA backend; no separate build option is
+required. Upstream GGML and non-CUDA backends do not support it. A system GGML
+must provide the matching patched API and CUDA implementation. Tensor-parallel
+row splitting is not supported; layer splitting requires supported devices.
+
+## Use
+
+Add `--sol-attn` to an existing generation command:
+
+```sh
+sd-cli ... --sol-attn
+sd-cli ... --sol-attn --sol-attn-tau 1.0
+```
+
+The default threshold coefficient is `1.0`. Larger coefficients select fewer
+blocks for exact attention. The coefficient must be finite; zero does not mean
+dense attention. Omit `--sol-attn` to disable the feature.
+
+The native kernel supports unmasked, noncausal attention with head dimension
+128, equal Q/K/V sequence lengths and head counts, and multiple batches. Other
+attention operations fall back to FlashAttention when available, then ordinary
+attention. Existing attention scaling overrides remain effective. `--fa` and
+`--diffusion-fa` may be used together with Sol-Attn; `--sage-attn` is mutually
+exclusive. Text encoders and VAEs retain their existing attention selection.
+
+Initialization reports an error if the requested diffusion backend cannot run
+Sol-Attn. Graph logs report the number of Sol-Attn and FlashAttention nodes and
+warn when no Sol-Attn nodes are selected. CUDA execution errors are not silently
+converted into dense attention.
+
+This is approximate attention. Validate quality and end-to-end speed with the
+same prompt, seed, dimensions, frame count, and sampling settings. Include
+packing, preprocessing, offload, and decode time in comparisons. Short sequences
+may not benefit. Upstream combined pipeline speedups are not measurements of
+this native kernel. Exact-covariance thresholds, text sinks, Morton ordering,
+and step/layer schedules are not implemented.
+
+## Validation
+
+On an RTX 4090 with CUDA 12.4, Wan 2.1 T2V 1.3B was tested at 832x480,
+33 frames, 20 Euler steps, seed 42, CFG 6, and flow shift 3, using the prompt
+`a lovely cat` and the same negative prompt for every run:
+
+| Attention | Sampling time | Total process time |
+| --- | ---: | ---: |
+| FlashAttention | 45.73 s | 74.63 s |
+| Sol-Attn, tau 1 | 37.17 s | 66.20 s |
+| Sol-Attn, tau 0 | 40.66 s | 68.50 s |
+
+These are single-run measurements. The graph selected 30 Sol-Attn nodes and
+30 FlashAttention nodes. At tau 1, sampled video frames showed washed-out
+colors and reduced detail. Tau 0 improved clarity in this example, but still
+changed the composition. Neither setting guarantees the baseline's quality.
+For this Wan command, `--sol-attn --sol-attn-tau 0` is a more conservative
+starting point. In the one-frame case, tau 1 increased warm sampling time from
+0.140 to 0.148 seconds per step.
+
+Validation also covered 15 numerical reference cases, 11 layout/scaling/fallback
+cases, CUDA memory checking, and 36 existing SageAttention regression cases.
+CLI and server CUDA builds and the upstream GGML CPU library build passed.
+Other GPU architectures, multi-GPU execution, and other models have not been
+tested.
+
+## Library API
+
+Configure Sol-Attn in `sd_ctx_params_t` before creating the context:
+
+```cpp
+sd_ctx_params_t params;
+sd_ctx_params_init(&params);
+// Set model paths and other context options here.
+params.sol_attn = true;
+params.sol_attn_tau = 1.0f;
+sd_ctx_t* ctx = new_sd_ctx(&params);
+```
+
+`sd_ctx_params_init` defaults `sol_attn` to false and `sol_attn_tau` to 1.0.
+`new_sd_ctx` returns null for a nonfinite threshold, unavailable requested
+backends, or a conflict with SageAttention. The context owns a copy of these
+settings; changing the input structure after creation does not reconfigure it.
+Applications must be rebuilt against the updated `sd_ctx_params_t` definition.

docs/troubleshooting.md

@@ -1,14 +1,5 @@
 # Troubleshooting
 
-## Video model used in image generation mode
-
-If generation reports that a model cannot be run with `generate_image()`, add
-`--mode vid_gen` to the CLI command. `--video-frames` alone does not select video
-mode. Video models require this mode even when generating a single frame.
-Library callers must use `generate_video()` for these models; use
-`sd_ctx_supports_image_generation()` and `sd_ctx_supports_video_generation()` to
-check the available generation modes.
-
 ## Completely black or white images or videos / NaNs
 
 Some ggml backends can encounter numerical overflow during inference, producing

docs/wan.md

@@ -1,7 +1,5 @@
 # How to Use
 
-Wan models require `-M vid_gen`, including single-frame generation. `--video-frames` alone does not select video mode. Library callers must use `generate_video()` instead of `generate_image()`.
-
 ## Download weights
 
 - Download Wan

examples/cli/README.md

@@ -14,12 +14,6 @@ equivalent to `--log-level verbose`. If repeated, the last logging option wins.
 For direct image repair or automatic post-generation YOLOv8 detection followed by cropped inpainting, see
 [ADetailer](../../docs/adetailer.md).
 
-Use repeatable `--image-preprocess` rules to select resizing, cropping, padding,
-and resampling separately for each image input. Add `canny=true` to any input
-rule for edge detection. See
-[Image preprocessing](../../docs/image_preprocessing.md) for input selectors,
-input defaults, downstream model processing, mask alignment, and examples.
-
 Metadata mode inspects PNG/JPEG container metadata without loading any model:
 
 ```bash
@@ -31,3 +25,7 @@ Metadata mode inspects PNG/JPEG container metadata without loading any model:
 
 For completely black or white images or videos, NaNs, and the `--linear-scale` /
 `--attn-scale` workaround, see [Troubleshooting](../../docs/troubleshooting.md).
+
+For native CUDA sparse attention in the diffusion model, use `--sol-attn`.
+See [Sol-Attn](../../docs/sol_attention.md) for requirements, supported shapes,
+and the `--sol-attn-tau` threshold coefficient.

examples/cli/main.cpp

@@ -41,6 +41,7 @@ struct SDCliParams {
     std::string metadata_format = "text";
 
     sd_log_level_t log_level = SD_LOG_INFO;
+    bool canny_preprocess    = false;
     bool convert_name        = false;
 
     preview_t preview_method = PREVIEW_NONE;
@@ -106,6 +107,10 @@ struct SDCliParams {
         };
 
         options.bool_options = {
+            {"",
+             "--canny",
+             "apply canny preprocessor (edge detection)",
+             true, &canny_preprocess},
             {"",
              "--convert-name",
              "convert tensor name (for convert mode)",
@@ -263,6 +268,7 @@ struct SDCliParams {
             << "  metadata_format: \"" << metadata_format << "\",\n"
             << "  log_level: " << log_level_name(log_level) << ",\n"
             << "  color: " << (color ? "true" : "false") << ",\n"
+            << "  canny_preprocess: " << (canny_preprocess ? "true" : "false") << ",\n"
             << "  convert_name: " << (convert_name ? "true" : "false") << ",\n"
             << "  preview_method: " << previews_str[preview_method] << ",\n"
             << "  preview_interval: " << preview_interval << ",\n"
@@ -322,7 +328,9 @@ void sd_log_cb(enum sd_log_level_t level, const char* log, void* data) {
 
 bool load_images_from_dir(const std::string dir,
                           std::vector<SDImageOwner>& images,
-                          int max_image_num = 0) {
+                          int expected_width  = 0,
+                          int expected_height = 0,
+                          int max_image_num   = 0) {
     if (!fs::exists(dir) || !fs::is_directory(dir)) {
         LOG_ERROR("'%s' is not a valid directory\n", dir.c_str());
         return false;
@@ -349,8 +357,7 @@ bool load_images_from_dir(const std::string dir,
             LOG_VERBOSE("load image %zu from '%s'", images.size(), path.c_str());
             int width             = 0;
             int height            = 0;
-            int loaded_channel    = 0;
-            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, 0, 0);
+            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, expected_width, expected_height);
             if (image_buffer == nullptr) {
                 LOG_ERROR("load image from '%s' failed", path.c_str());
                 return false;
@@ -358,7 +365,7 @@ bool load_images_from_dir(const std::string dir,
 
             images.emplace_back(sd_image_t{(uint32_t)width,
                                            (uint32_t)height,
-                                           (uint32_t)loaded_channel,
+                                           3,
                                            image_buffer});
 
             if (max_image_num > 0 && static_cast<int>(images.size()) >= max_image_num) {
@@ -646,8 +653,8 @@ int main(int argc, const char* argv[]) {
     SDContextParams ctx_params;
     SDGenerationParams gen_params;
 
-    sd_set_log_callback(sd_log_cb, (void*)&cli_params);
     parse_args(argc, argv, cli_params, ctx_params, gen_params);
+    sd_set_log_callback(sd_log_cb, (void*)&cli_params);
 
     if (cli_params.mode == METADATA) {
         MetadataReadOptions options;
@@ -743,8 +750,16 @@ int main(int argc, const char* argv[]) {
 
     auto load_image_and_update_size = [&](const std::string& path,
                                           SDImageOwner& image,
+                                          bool resize_image    = true,
                                           int expected_channel = 3) -> bool {
-        if (!load_sd_image_from_file(image.put(), path.c_str(), 0, 0, expected_channel)) {
+        int expected_width  = 0;
+        int expected_height = 0;
+        if (resize_image && gen_params.width_and_height_are_set()) {
+            expected_width  = gen_params.width;
+            expected_height = gen_params.height;
+        }
+
+        if (!load_sd_image_from_file(image.put(), path.c_str(), expected_width, expected_height, expected_channel)) {
             LOG_ERROR("load image from '%s' failed", path.c_str());
             return false;
         }
@@ -766,8 +781,7 @@ int main(int argc, const char* argv[]) {
     };
 
     if (gen_params.init_image_path.size() > 0) {
-        const bool native_init = cli_params.mode == IMG_GEN || cli_params.mode == ADETAILER;
-        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, native_init ? 0 : 3)) {
+        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image)) {
             return 1;
         }
     }
@@ -781,8 +795,8 @@ int main(int argc, const char* argv[]) {
     if (gen_params.ref_image_paths.size() > 0) {
         gen_params.ref_images.clear();
         for (auto& path : gen_params.ref_image_paths) {
-            SDImageOwner ref_image({0, 0, 0, nullptr});
-            if (!load_image_and_update_size(path, ref_image, 0)) {
+            SDImageOwner ref_image({0, 0, 3, nullptr});
+            if (!load_image_and_update_size(path, ref_image, false)) {
                 return 1;
             }
             gen_params.ref_images.push_back(std::move(ref_image));
@@ -823,22 +837,41 @@ int main(int argc, const char* argv[]) {
     if (gen_params.mask_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.mask_image.put(),
                                      gen_params.mask_image_path.c_str(),
-                                     0,
-                                     0,
+                                     gen_params.get_resolved_width(),
+                                     gen_params.get_resolved_height(),
                                      1)) {
             LOG_ERROR("load image from '%s' failed", gen_params.mask_image_path.c_str());
             return 1;
         }
+    } else {
+        sd_image_t generated_mask = {0, 0, 1, nullptr};
+        generated_mask.data       = (uint8_t*)malloc(gen_params.get_resolved_width() * gen_params.get_resolved_height());
+        if (generated_mask.data == nullptr) {
+            LOG_ERROR("malloc mask image failed");
+            return 1;
+        }
+        generated_mask.width  = gen_params.get_resolved_width();
+        generated_mask.height = gen_params.get_resolved_height();
+        memset(generated_mask.data, 255, gen_params.get_resolved_width() * gen_params.get_resolved_height());
+        gen_params.mask_image.reset(generated_mask);
     }
 
     if (gen_params.control_image_path.size() > 0) {
         if (!load_sd_image_from_file(gen_params.control_image.put(),
                                      gen_params.control_image_path.c_str(),
-                                     0,
-                                     0)) {
+                                     gen_params.get_resolved_width(),
+                                     gen_params.get_resolved_height())) {
             LOG_ERROR("load image from '%s' failed", gen_params.control_image_path.c_str());
             return 1;
         }
+        if (cli_params.canny_preprocess) {  // apply preprocessor
+            preprocess_canny(gen_params.control_image.get(),
+                             0.08f,
+                             0.08f,
+                             0.8f,
+                             1.0f,
+                             false);
+        }
     }
 
     if (gen_params.ip_adapter_image_path.size() > 0) {
@@ -855,6 +888,8 @@ int main(int argc, const char* argv[]) {
         gen_params.control_frames.clear();
         if (!load_images_from_dir(gen_params.control_video_path,
                                   gen_params.control_frames,
+                                  gen_params.get_resolved_width(),
+                                  gen_params.get_resolved_height(),
                                   gen_params.video_frames)) {
             return 1;
         }
@@ -863,7 +898,10 @@ int main(int argc, const char* argv[]) {
     if (!gen_params.pm_id_images_dir.empty()) {
         gen_params.pm_id_images.clear();
         if (!load_images_from_dir(gen_params.pm_id_images_dir,
-                                  gen_params.pm_id_images)) {
+                                  gen_params.pm_id_images,
+                                  0,
+                                  0,
+                                  0)) {
             return 1;
         }
     }

examples/common/common.cpp

@@ -622,6 +622,10 @@ ArgOptions SDContextParams::get_options() {
          "--sage-attn",
          "use native CUDA SageAttention in the diffusion model, with flash/default attention fallback",
          true, &sage_attn},
+        {"",
+         "--sol-attn",
+         "use native CUDA Sol-Attn in the diffusion model, with flash/default attention fallback",
+         true, &sol_attn},
         {"",
          "--diffusion-conv-direct",
          "use ggml_conv2d_direct in the diffusion model",
@@ -719,6 +723,8 @@ ArgOptions SDContextParams::get_options() {
         return 1;
     };
 
+    options.float_options.push_back({"", "--sol-attn-tau", "Sol-Attn routing threshold coefficient (default: 1; higher selects fewer exact blocks)", &sol_attn_tau});
+
     options.manual_options = {
         {"",
          "--linear-scale",
@@ -822,6 +828,14 @@ bool SDContextParams::resolve(SDMode mode) {
 }
 
 bool SDContextParams::validate(SDMode mode) {
+    if (sol_attn && sage_attn) {
+        LOG_ERROR("--sol-attn and --sage-attn cannot be enabled together");
+        return false;
+    }
+    if (!std::isfinite(sol_attn_tau)) {
+        LOG_ERROR("--sol-attn-tau must be finite");
+        return false;
+    }
     if (mode == CONVERT) {
         const bool has_convert_input = model_path.length() != 0 ||
                                        clip_l_path.length() != 0 ||
@@ -943,6 +957,8 @@ std::string SDContextParams::to_string() const {
         << "  flash_attn: " << (flash_attn ? "true" : "false") << ",\n"
         << "  diffusion_flash_attn: " << (diffusion_flash_attn ? "true" : "false") << ",\n"
         << "  sage_attn: " << (sage_attn ? "true" : "false") << ",\n"
+        << "  sol_attn: " << (sol_attn ? "true" : "false") << ",\n"
+        << "  sol_attn_tau: " << sol_attn_tau << ",\n"
         << "  linear_scale: " << linear_scale << ",\n"
         << "  attn_scale: " << attn_scale << ",\n"
         << "  diffusion_conv_direct: " << (diffusion_conv_direct ? "true" : "false") << ",\n"
@@ -1001,6 +1017,8 @@ sd_ctx_params_t SDContextParams::to_sd_ctx_params_t(bool taesd_preview) {
     sd_ctx_params.flash_attn                      = flash_attn;
     sd_ctx_params.diffusion_flash_attn            = diffusion_flash_attn;
     sd_ctx_params.sage_attn                       = sage_attn;
+    sd_ctx_params.sol_attn                        = sol_attn;
+    sd_ctx_params.sol_attn_tau                    = sol_attn_tau;
     sd_ctx_params.linear_scale                    = linear_scale;
     sd_ctx_params.attn_scale                      = attn_scale;
     sd_ctx_params.tae_preview_only                = taesd_preview;
@@ -1128,9 +1146,6 @@ ArgOptions SDGenerationParams::get_options() {
          "Key-value list to set up the way the reference images are processed (empty = auto-detect from model weigths)",
          (int)',',
          &ref_image_args},
-        {"", "--image-preprocess",
-         "Image preprocessing rule: target=init|end|mask|control|ref|ip-adapter|id|control-frame,index=N,mode=auto|none|stretch|crop|crop-resize|fit-pad,filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos,antialias=auto|true|false,width=W,height=H,anchor=center|top|bottom|left|right,pad_color=#RRGGBB[AA],canny=true|false. Repeat for multiple rules.",
-         (int)';', &image_preprocess},
     };
 
     options.int_options = {
@@ -1311,6 +1326,11 @@ ArgOptions SDGenerationParams::get_options() {
          "automatically increase the indices of references images based on the order they are listed (starting with 1).",
          true,
          &increase_ref_index},
+        {"",
+         "--disable-auto-resize-ref-image",
+         "disable auto resize of ref images",
+         false,
+         &auto_resize_ref_image},
         {"",
          "--circular",
          "enable circular padding on both axes for tileable output",
@@ -1752,7 +1772,7 @@ ArgOptions SDGenerationParams::get_options() {
          on_scm_policy_arg},
         {"",
          "--vae-tile-size",
-         "tile size for vae tiling in latent units, not image pixels, format [X]x[Y] (default: 32x32)",
+         "tile size for vae tiling, format [X]x[Y] (default: 32x32)",
          on_tile_size_arg},
         {"",
          "--vae-relative-tile-size",
@@ -1846,28 +1866,28 @@ bool decode_base64_image(const std::string& encoded_input,
         return false;
     }
 
-    int decoded_width    = 0;
-    int decoded_height   = 0;
-    int resolved_channel = target_channels;
-    uint8_t* raw_data    = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
-                                                  static_cast<int>(image_bytes.size()),
-                                                  decoded_width,
-                                                  decoded_height,
-                                                  resolved_channel,
-                                                  expected_width,
-                                                  expected_height,
-                                                  target_channels);
+    int decoded_width  = 0;
+    int decoded_height = 0;
+    uint8_t* raw_data  = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
+                                                static_cast<int>(image_bytes.size()),
+                                                decoded_width,
+                                                decoded_height,
+                                                expected_width,
+                                                expected_height,
+                                                target_channels);
     if (raw_data == nullptr) {
         return false;
     }
 
-    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)resolved_channel, raw_data});
+    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)target_channels, raw_data});
     return true;
 }
 
 static bool parse_image_json_field(const json& parent,
                                    const char* key,
                                    int channels,
+                                   int expected_width,
+                                   int expected_height,
                                    SDImageOwner& out_image) {
     if (!parent.contains(key)) {
         return true;
@@ -1879,12 +1899,14 @@ static bool parse_image_json_field(const json& parent,
     if (!parent.at(key).is_string()) {
         return false;
     }
-    return decode_base64_image(parent.at(key).get<std::string>(), channels, 0, 0, out_image);
+    return decode_base64_image(parent.at(key).get<std::string>(), channels, expected_width, expected_height, out_image);
 }
 
 static bool parse_image_array_json_field(const json& parent,
                                          const char* key,
                                          int channels,
+                                         int expected_width,
+                                         int expected_height,
                                          std::vector<SDImageOwner>& out_images) {
     if (!parent.contains(key)) {
         return true;
@@ -1903,7 +1925,7 @@ static bool parse_image_array_json_field(const json& parent,
             return false;
         }
         SDImageOwner image;
-        if (!decode_base64_image(item.get<std::string>(), channels, 0, 0, image)) {
+        if (!decode_base64_image(item.get<std::string>(), channels, expected_width, expected_height, image)) {
             return false;
         }
         out_images.push_back(std::move(image));
@@ -2002,29 +2024,6 @@ static bool resolve_model_file_from_dir(const std::string& model_name,
     return false;
 }
 
-bool SDGenerationParams::parse_image_preprocess_json(const std::string& json_str) {
-    const auto value = json::parse(json_str, nullptr, false);
-    std::string rules;
-    if (value.is_string()) {
-        rules = value.get<std::string>();
-    } else if (value.is_array()) {
-        for (const auto& item : value) {
-            if (!item.is_string()) {
-                LOG_ERROR("image_preprocess must contain rule strings");
-                return false;
-            }
-            if (!rules.empty())
-                rules += ";";
-            rules += item.get<std::string>();
-        }
-    } else {
-        LOG_ERROR("image_preprocess must be a string or array of strings");
-        return false;
-    }
-    image_preprocess = std::move(rules);
-    return true;
-}
-
 bool SDGenerationParams::from_json_str(
     const std::string& json_str,
     const std::function<std::string(const std::string&)>& lora_path_resolver) {
@@ -2036,9 +2035,6 @@ bool SDGenerationParams::from_json_str(
         return false;
     }
 
-    if (j.contains("image_preprocess") && !parse_image_preprocess_json(j["image_preprocess"].dump()))
-        return false;
-
     auto load_if_exists = [&](const char* key, auto& out) {
         if (j.contains(key)) {
             using T = std::decay_t<decltype(out)>;
@@ -2076,7 +2072,6 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("cache_mode", cache_mode);
     load_if_exists("cache_option", cache_option);
     load_if_exists("scm_mask", scm_mask);
-    load_if_exists("ref_image_args", ref_image_args);
 
     load_if_exists("clip_skip", clip_skip);
     load_if_exists("width", width);
@@ -2094,6 +2089,7 @@ bool SDGenerationParams::from_json_str(
     load_if_exists("moe_boundary", moe_boundary);
     load_if_exists("vace_strength", vace_strength);
 
+    load_if_exists("auto_resize_ref_image", auto_resize_ref_image);
     load_if_exists("increase_ref_index", increase_ref_index);
     load_if_exists("embed_image_metadata", embed_image_metadata);
 
@@ -2237,23 +2233,32 @@ bool SDGenerationParams::from_json_str(
         LOG_ERROR("invalid lora");
         return false;
     }
-    auto load_image = [&](const char* key, int channels, SDImageOwner& image) {
-        if (!parse_image_json_field(j, key, channels, image)) {
-            LOG_ERROR("invalid %s", key);
-            return false;
-        }
-        return true;
-    };
-    if (!load_image("init_image", 0, init_image) ||
-        !load_image("end_image", 3, end_image) ||
-        !load_image("mask_image", 1, mask_image) ||
-        !load_image("control_image", 3, control_image) ||
-        !load_image("ip_adapter_image", 3, ip_adapter_image)) {
+    if (!parse_image_json_field(j, "init_image", 3, width, height, init_image)) {
+        LOG_ERROR("invalid init_image");
         return false;
     }
-    if (!parse_image_array_json_field(j, "ref_images", 0, ref_images) ||
-        !parse_image_array_json_field(j, "control_frames", 3, control_frames)) {
-        LOG_ERROR("invalid input image array");
+    if (!parse_image_json_field(j, "end_image", 3, width, height, end_image)) {
+        LOG_ERROR("invalid end_image");
+        return false;
+    }
+    if (!parse_image_array_json_field(j, "ref_images", 3, width, height, ref_images)) {
+        LOG_ERROR("invalid ref_images");
+        return false;
+    }
+    if (!parse_image_array_json_field(j, "control_frames", 3, width, height, control_frames)) {
+        LOG_ERROR("invalid control_frames");
+        return false;
+    }
+    if (!parse_image_json_field(j, "mask_image", 1, width, height, mask_image)) {
+        LOG_ERROR("invalid mask_image");
+        return false;
+    }
+    if (!parse_image_json_field(j, "control_image", 3, width, height, control_image)) {
+        LOG_ERROR("invalid control_image");
+        return false;
+    }
+    if (!parse_image_json_field(j, "ip_adapter_image", 3, width, height, ip_adapter_image)) {
+        LOG_ERROR("invalid ip_adapter_image");
         return false;
     }
 
@@ -2497,10 +2502,6 @@ bool SDGenerationParams::resolve(const std::string& lora_model_dir, const std::s
 }
 
 bool SDGenerationParams::validate(SDMode mode) {
-    if (!image_preprocess.empty() && mode != IMG_GEN && mode != VID_GEN) {
-        LOG_ERROR("--image-preprocess requires img_gen or vid_gen mode");
-        return false;
-    }
     if (batch_count <= 0) {
         LOG_ERROR("error: batch_count must be greater than 0");
         return false;
@@ -2676,6 +2677,14 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
         pulid_id_weight,
     };
 
+    if (!auto_resize_ref_image) {
+        if (!ref_image_args.empty()) {
+            ref_image_args += ",";
+        }
+        ref_image_args += "resize_before_vae=0";
+        LOG_WARN("Notice: --disable-auto-resize-ref-image is deprecated. Use --ref-image-args \"resize_before_vae=off\" instead.");
+    }
+
     if (increase_ref_index) {
         if (!ref_image_args.empty()) {
             ref_image_args += ",";
@@ -2723,7 +2732,6 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
-    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2826,7 +2834,6 @@ sd_vid_gen_params_t SDGenerationParams::to_sd_vid_gen_params_t() {
     params.hires.custom_sigmas_count = static_cast<int>(hires_custom_sigmas.size());
     params.circular_x                = circular || circular_x;
     params.circular_y                = circular || circular_y;
-    params.image_preprocess          = {image_preprocess.c_str()};
     return params;
 }
 
@@ -2883,8 +2890,7 @@ std::string SDGenerationParams::to_string() const {
         << "  ref_video_audio_paths: " << vec_str_to_string(ref_video_audio_paths) << ",\n"
         << "  ref_audio_paths: " << vec_str_to_string(ref_audio_paths) << ",\n"
         << "  control_video_path: \"" << control_video_path << "\",\n"
-        << "  image_preprocess: " << image_preprocess << ",\n"
-        << "  ref_image_args: " << ref_image_args << ",\n"
+        << "  auto_resize_ref_image: " << (auto_resize_ref_image ? "true" : "false") << ",\n"
         << "  increase_ref_index: " << (increase_ref_index ? "true" : "false") << ",\n"
         << "  pm_id_images_dir: \"" << pm_id_images_dir << "\",\n"
         << "  pm_id_embed_path: \"" << pm_id_embed_path << "\",\n"
@@ -3035,13 +3041,12 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
     set_json_basename_if_not_empty(models, "control_net", ctx_params.control_net_path);
     root["models"] = std::move(models);
 
-    root["clip_skip"]           = gen_params.clip_skip;
-    root["strength"]            = gen_params.strength;
-    root["control_strength"]    = gen_params.control_strength;
-    root["ip_adapter_strength"] = gen_params.ip_adapter_strength;
-    root["ref_image_args"]      = gen_params.ref_image_args;
-    root["image_preprocess"]    = gen_params.image_preprocess;
-    root["increase_ref_index"]  = gen_params.increase_ref_index;
+    root["clip_skip"]             = gen_params.clip_skip;
+    root["strength"]              = gen_params.strength;
+    root["control_strength"]      = gen_params.control_strength;
+    root["ip_adapter_strength"]   = gen_params.ip_adapter_strength;
+    root["auto_resize_ref_image"] = gen_params.auto_resize_ref_image;
+    root["increase_ref_index"]    = gen_params.increase_ref_index;
     if (mode == VID_GEN) {
         root["video"] = {
             {"frame_count", gen_params.video_frames},

examples/common/common.h

@@ -171,6 +171,8 @@ struct SDContextParams {
     bool flash_attn            = false;
     bool diffusion_flash_attn  = false;
     bool sage_attn             = false;
+    bool sol_attn              = false;
+    float sol_attn_tau         = 1.f;
     bool diffusion_conv_direct = false;
     bool vae_conv_direct       = false;
 
@@ -200,17 +202,18 @@ struct SDGenerationParams {
     std::string ad_prompt;
     std::string ad_negative_prompt;
     std::string extra_ad_args;
-    int clip_skip             = -1;  // <= 0 represents unspecified
-    int width                 = -1;
-    int height                = -1;
-    int batch_count           = 1;
-    int qwen_image_layers     = 3;
-    int64_t seed              = 42;
-    float strength            = 0.75f;
-    float control_strength    = 0.9f;
-    float ip_adapter_strength = 1.0f;
-    bool increase_ref_index   = false;
-    bool embed_image_metadata = true;
+    int clip_skip              = -1;  // <= 0 represents unspecified
+    int width                  = -1;
+    int height                 = -1;
+    int batch_count            = 1;
+    int qwen_image_layers      = 3;
+    int64_t seed               = 42;
+    float strength             = 0.75f;
+    float control_strength     = 0.9f;
+    float ip_adapter_strength  = 1.0f;
+    bool auto_resize_ref_image = true;
+    bool increase_ref_index    = false;
+    bool embed_image_metadata  = true;
 
     std::string init_image_path;
     std::string end_image_path;
@@ -246,7 +249,6 @@ struct SDGenerationParams {
     std::string extra_tiling_args;
 
     std::string ref_image_args;
-    std::string image_preprocess;
 
     std::string pm_id_images_dir;
     std::string pm_id_embed_path;
@@ -310,7 +312,6 @@ struct SDGenerationParams {
     ArgOptions get_options();
     bool from_json_str(const std::string& json_str,
                        const std::function<std::string(const std::string&)>& lora_path_resolver = {});
-    bool parse_image_preprocess_json(const std::string& json_str);
     bool initialize_cache_params();
     void extract_and_remove_lora(const std::string& lora_model_dir);
     bool width_and_height_are_set() const;

examples/common/media_io.cpp

@@ -261,10 +261,6 @@ uint8_t* decode_webp_image_to_buffer(const uint8_t* data,
     height               = features.height;
     source_channel_count = features.has_alpha ? 4 : 3;
 
-    if (expected_channel == 0) {
-        expected_channel = source_channel_count;
-    }
-
     const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
 
     if (expected_channel == 1) {
@@ -485,8 +481,7 @@ uint8_t* load_image_common(bool from_memory,
                            int& height,
                            int expected_width,
                            int expected_height,
-                           int expected_channel,
-                           int& out_channel) {
+                           int expected_channel) {
     const char* image_path;
     FreeUniquePtr<uint8_t> image_buffer;
     int source_channel_count = 0;
@@ -543,32 +538,6 @@ uint8_t* load_image_common(bool from_memory,
         LOG_ERROR("load image from '%s' failed", image_path);
         return nullptr;
     }
-    if (expected_channel == 0) {
-        expected_channel = source_channel_count == 2 ? 4 : (source_channel_count == 1 ? 3 : source_channel_count);
-        if (expected_channel != source_channel_count) {
-            FreeUniquePtr<uint8_t> promoted((uint8_t*)malloc((size_t)width * height * expected_channel));
-            if (promoted == nullptr) {
-                LOG_ERROR("error: allocate memory for channel promotion, image_path = %s", image_path);
-                return nullptr;
-            }
-            const size_t pixel_count = (size_t)width * (size_t)height;
-            for (size_t i = 0; i < pixel_count; ++i) {
-                if (source_channel_count == 1) {
-                    promoted.get()[i * 3 + 0] = image_buffer.get()[i];
-                    promoted.get()[i * 3 + 1] = image_buffer.get()[i];
-                    promoted.get()[i * 3 + 2] = image_buffer.get()[i];
-                } else {
-                    promoted.get()[i * 4 + 0] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 1] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 2] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 3] = image_buffer.get()[i * 2 + 1];
-                }
-            }
-            image_buffer         = std::move(promoted);
-            source_channel_count = expected_channel;
-        }
-    }
-    // stb reports the source channel count even when it converts the output.
     if (source_channel_count < expected_channel) {
         fprintf(stderr,
                 "the number of channels for the input image must be >= %d,"
@@ -628,7 +597,7 @@ uint8_t* load_image_common(bool from_memory,
         }
         stbir_resize(image_buffer.get(), width, height, 0,
                      resized_image_buffer.get(), expected_width, expected_height, 0, STBIR_TYPE_UINT8,
-                     expected_channel, expected_channel == 4 ? 3 : STBIR_ALPHA_CHANNEL_NONE, 0,
+                     expected_channel, STBIR_ALPHA_CHANNEL_NONE, 0,
                      STBIR_EDGE_CLAMP, STBIR_EDGE_CLAMP,
                      STBIR_FILTER_BOX, STBIR_FILTER_BOX,
                      STBIR_COLORSPACE_SRGB, nullptr);
@@ -636,7 +605,6 @@ uint8_t* load_image_common(bool from_memory,
         height       = expected_height;
         image_buffer = std::move(resized_image_buffer);
     }
-    out_channel = expected_channel;
     return image_buffer.release();
 }
 
@@ -809,11 +777,10 @@ bool write_image_to_file(const std::string& path,
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
-                              int& out_channel,
                               int expected_width,
                               int expected_height,
                               int expected_channel) {
-    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, out_channel);
+    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
 }
 
 bool load_sd_image_from_file(sd_image_t* image,
@@ -823,14 +790,13 @@ bool load_sd_image_from_file(sd_image_t* image,
                              int expected_channel) {
     int width;
     int height;
-    int resolved_channel = expected_channel;
-    image->data          = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, resolved_channel);
+    image->data = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
     if (image->data == nullptr) {
         return false;
     }
     image->width   = width;
     image->height  = height;
-    image->channel = resolved_channel;
+    image->channel = expected_channel;
     return true;
 }
 
@@ -838,11 +804,10 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
-                                int& out_channel,
                                 int expected_width,
                                 int expected_height,
                                 int expected_channel) {
-    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel, out_channel);
+    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel);
 }
 
 static void append_avi_metadata(std::vector<uint8_t>& data, const std::string& parameters) {

examples/common/media_io.h

@@ -32,12 +32,9 @@ bool write_image_to_file(const std::string& path,
                          const std::string& parameters = "",
                          int quality                   = 90);
 
-// expected_channel == 0 preserves native channels (grayscale -> RGB, gray+alpha -> RGBA).
-// out_channel receives the output channel count.
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
-                              int& out_channel,
                               int expected_width   = 0,
                               int expected_height  = 0,
                               int expected_channel = 3);
@@ -52,7 +49,6 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
-                                int& out_channel,
                                 int expected_width   = 0,
                                 int expected_height  = 0,
                                 int expected_channel = 3);

examples/server/api.md

@@ -148,19 +148,6 @@ Native extension fields:
 
 - any `sdcpp API` fields embedded through `sd_cpp_extra_args` inside `prompt`
 
-Uploaded images are decoded at their original dimensions. The first decoded
-image establishes the generation dimensions if `size` is omitted. Input
-geometry follows `image_preprocess`: references preserve their dimensions by
-default, while init and mask use the generation canvas preset.
-
-Reference encoding then follows model presets and `ref_image_args`. To skip
-input geometry for references and disable resizing before VAE encoding, include
-this in `prompt`:
-
-```text
-edit this image <sd_cpp_extra_args>{"image_preprocess":"target=ref,mode=none","ref_image_args":"resize_before_vae=false"}</sd_cpp_extra_args>
-```
-
 Response fields:
 
 | Field | Type | Notes |
@@ -539,7 +526,7 @@ LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEH
 | Field | Type |
 | --- | --- |
 | `batch_count` | `integer` |
-| `ref_image_args` | `string` |
+| `auto_resize_ref_image` | `boolean` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |
@@ -666,7 +653,7 @@ Example:
   "strength": 0.75,
   "seed": -1,
   "batch_count": 1,
-  "ref_image_args": "",
+  "auto_resize_ref_image": true,
   "increase_ref_index": false,
   "control_strength": 0.9,
   "ip_adapter_strength": 1.0,
@@ -741,17 +728,6 @@ Example:
 
 ### Image Encoding Rules
 
-Native image/video requests and SDAPI accept `image_preprocess` as a rule string
-or array of rule strings. OpenAI-compatible requests can supply it in
-`sd_cpp_extra_args`. See [Image preprocessing](../../docs/image_preprocessing.md)
-for one-time input geometry, native-resolution decoding, mask alignment, and
-`canny=true` for edge detection on any supported image input.
-
-Image generation also accepts `ref_image_args` as a string (for example,
-`"resize_before_vae=false"`) in native and SDAPI requests, or through
-`sd_cpp_extra_args` in OpenAI-compatible requests. It controls downstream
-reference encoding and is independent of input geometry rules.
-
 Any image field accepts:
 
 - a raw base64 string, or
@@ -759,15 +735,12 @@ Any image field accepts:
 
 Channel expectations:
 
-- `init_image`: native channels (3 or 4); alpha is preserved and applied per model
-- `ref_images[]`: native channels (3 or 4); alpha is preserved and applied per model
+- `init_image`: 3 channels
+- `ref_images[]`: 3 channels
 - `control_image`: 3 channels
 - `ip_adapter_image`: 3 channels
 - `mask_image`: 1 channel
 
-Models that support RGBA (e.g. Qwen-Image 2.1) use the alpha channel of `init_image`
-and `ref_images[]`. RGB-only models drop it, so sending RGBA is safe for every model.
-
 If omitted or null:
 
 - single-image fields map to an empty `sd_image_t`
@@ -787,8 +760,7 @@ Top-level scalar fields:
 | `strength` | `number` |
 | `seed` | `integer` |
 | `batch_count` | `integer` |
-| `ref_image_args` | `string` |
-| `image_preprocess` | `string \| array<string>` |
+| `auto_resize_ref_image` | `boolean` |
 | `increase_ref_index` | `boolean` |
 | `control_strength` | `number` |
 | `ip_adapter_strength` | `number` |

examples/server/main.cpp

@@ -76,9 +76,9 @@ int main(int argc, const char** argv) {
     SDSvrParams svr_params;
     SDContextParams ctx_params;
     SDGenerationParams default_gen_params;
+    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     sd_set_log_callback(sd_log_cb, (void*)&svr_params);
-    parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
 
     LOG_VERBOSE("version: %s", version_string().c_str());
     LOG_VERBOSE("%s", sd_get_system_info());

examples/server/routes_openai.cpp

@@ -157,46 +157,42 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.height      = height;
     request.gen_params.batch_count = n;
 
-    std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
     for (auto& bytes : images_bytes) {
-        int img_w            = 0;
-        int img_h            = 0;
-        int resolved_channel = 0;
-        uint8_t* raw_pixels  = load_image_from_memory(
-             reinterpret_cast<const char*>(bytes.data()),
-             static_cast<int>(bytes.size()),
-             img_w, img_h, resolved_channel,
-             0, 0,
-             0);
+        int img_w           = 0;
+        int img_h           = 0;
+        uint8_t* raw_pixels = load_image_from_memory(
+            reinterpret_cast<const char*>(bytes.data()),
+            static_cast<int>(bytes.size()),
+            img_w, img_h,
+            width, height, 3);
         if (raw_pixels == nullptr) {
             continue;
         }
 
-        const bool is_first_ref_image = request.gen_params.ref_images.empty();
-        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, (uint32_t)resolved_channel, raw_pixels});
+        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, 3, raw_pixels});
         request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
-
-        if (is_first_ref_image) {
-            request.gen_params.init_image = image_owner;
-            if (request.gen_params.init_image.get().data == nullptr) {
-                error_message = "could not allocate init image";
-                return false;
-            }
-        }
-
         request.gen_params.ref_images.push_back(std::move(image_owner));
     }
 
+    if (!request.gen_params.ref_images.empty()) {
+        request.gen_params.init_image = request.gen_params.ref_images.front();
+    }
+
     if (!mask_bytes.empty()) {
-        int mask_w       = 0;
-        int mask_h       = 0;
-        int mask_channel = 0;
+        int expected_width  = 0;
+        int expected_height = 0;
+        if (request.gen_params.width_and_height_are_set()) {
+            expected_width  = request.gen_params.width;
+            expected_height = request.gen_params.height;
+        }
+        int mask_w = 0;
+        int mask_h = 0;
 
         uint8_t* mask_raw = load_image_from_memory(
             reinterpret_cast<const char*>(mask_bytes.data()),
             static_cast<int>(mask_bytes.size()),
-            mask_w, mask_h, mask_channel,
-            0, 0, 1);
+            mask_w, mask_h,
+            expected_width, expected_height, 1);
         request.gen_params.mask_image.reset({(uint32_t)mask_w, (uint32_t)mask_h, 1, mask_raw});
         const sd_image_t& mask_image = request.gen_params.mask_image.get();
         request.gen_params.set_width_and_height_if_unset(mask_image.width, mask_image.height);
@@ -209,6 +205,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
         });
     }
 
+    std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
     if (!sd_cpp_extra_args_str.empty() && !request.gen_params.from_json_str(sd_cpp_extra_args_str)) {
         error_message = "invalid sd_cpp_extra_args";
         return false;

examples/server/routes_sdapi.cpp

@@ -80,6 +80,17 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) {
     return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT;
 }
 
+static void assign_solid_mask(SDImageOwner& mask_owner, int width, int height) {
+    const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
+    uint8_t* raw_mask        = static_cast<uint8_t*>(malloc(pixel_count));
+    if (raw_mask == nullptr) {
+        mask_owner.reset({0, 0, 1, nullptr});
+        return;
+    }
+    std::memset(raw_mask, 255, pixel_count);
+    mask_owner.reset({(uint32_t)width, (uint32_t)height, 1, raw_mask});
+}
+
 static bool build_sdapi_img_gen_request(const json& j,
                                         ServerRuntime& runtime,
                                         bool img2img,
@@ -182,25 +193,15 @@ static bool build_sdapi_img_gen_request(const json& j,
         }
     }
 
-    if (j.contains("ref_image_args")) {
-        if (!j["ref_image_args"].is_string()) {
-            error_message = "ref_image_args must be a string";
-            return false;
-        }
-        request.gen_params.ref_image_args = j["ref_image_args"].get<std::string>();
-    }
-
-    if (j.contains("image_preprocess") && !request.gen_params.parse_image_preprocess_json(j["image_preprocess"].dump())) {
-        error_message = "invalid image_preprocess";
-        return false;
-    }
-
     if (img2img) {
+        const int expected_width  = request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0;
+        const int expected_height = request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0;
+
         if (j.contains("init_images") && j["init_images"].is_array() && !j["init_images"].empty()) {
             if (decode_base64_image(j["init_images"][0].get<std::string>(),
-                                    0,
-                                    0,
-                                    0,
+                                    3,
+                                    expected_width,
+                                    expected_height,
                                     request.gen_params.init_image)) {
                 const sd_image_t& image = request.gen_params.init_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -210,8 +211,8 @@ static bool build_sdapi_img_gen_request(const json& j,
         if (j.contains("mask") && j["mask"].is_string()) {
             if (decode_base64_image(j["mask"].get<std::string>(),
                                     1,
-                                    0,
-                                    0,
+                                    expected_width,
+                                    expected_height,
                                     request.gen_params.mask_image)) {
                 const sd_image_t& image = request.gen_params.mask_image.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -224,7 +225,9 @@ static bool build_sdapi_img_gen_request(const json& j,
                 }
             }
         } else {
-            request.gen_params.mask_image.reset({0, 0, 1, nullptr});
+            const int resolved_width  = request.gen_params.get_resolved_width();
+            const int resolved_height = request.gen_params.get_resolved_height();
+            assign_solid_mask(request.gen_params.mask_image, resolved_width, resolved_height);
         }
 
         float denoising_strength = j.value("denoising_strength", -1.f);
@@ -240,8 +243,9 @@ static bool build_sdapi_img_gen_request(const json& j,
             }
             SDImageOwner image_owner;
             if (decode_base64_image(extra_image.get<std::string>(),
-                                    0,
-                                    0, 0,
+                                    3,
+                                    request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0,
+                                    request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0,
                                     image_owner)) {
                 const sd_image_t& image = image_owner.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);

examples/server/routes_sdcpp.cpp

@@ -127,8 +127,7 @@ static json make_img_gen_defaults_json(const SDGenerationParams& defaults, const
         {"seed", defaults.seed},
         {"batch_count", defaults.batch_count},
         {"qwen_image_layers", defaults.qwen_image_layers},
-        {"ref_image_args", defaults.ref_image_args},
-        {"image_preprocess", defaults.image_preprocess},
+        {"auto_resize_ref_image", defaults.auto_resize_ref_image},
         {"increase_ref_index", defaults.increase_ref_index},
         {"control_strength", defaults.control_strength},
         {"ip_adapter_strength", defaults.ip_adapter_strength},
@@ -154,7 +153,6 @@ static json make_vid_gen_defaults_json(const SDGenerationParams& defaults, const
         {"strength", defaults.strength},
         {"seed", defaults.seed},
         {"video_frames", defaults.video_frames},
-        {"image_preprocess", defaults.image_preprocess},
         {"fps", defaults.fps},
         {"moe_boundary", defaults.moe_boundary},
         {"vace_strength", defaults.vace_strength},

include/stable-diffusion.h

@@ -247,6 +247,8 @@ typedef struct {
     float attn_scale;                // Override flash-attention K/V scaling; 0 keeps the model default
     const char* tokenizer;           // tokenizer.json path or main=FILE,clip-l=FILE,clip-g=FILE assignments; required for PiD and Lens
     bool sage_attn;
+    bool sol_attn;
+    float sol_attn_tau;
 } sd_ctx_params_t;
 
 typedef struct {
@@ -263,11 +265,6 @@ typedef struct {
     uint8_t* data;
 } sd_image_t;
 
-typedef struct {
-    // Semicolon-separated target=...,key=value rules. NULL preserves defaults.
-    const char* rules;
-} sd_image_preprocess_params_t;
-
 typedef struct {
     sd_image_t* frames;
     int frame_count;
@@ -415,7 +412,6 @@ typedef struct {
     int qwen_image_layers;
     bool circular_x;
     bool circular_y;
-    sd_image_preprocess_params_t image_preprocess;
 } sd_img_gen_params_t;
 
 typedef struct {
@@ -449,7 +445,6 @@ typedef struct {
     sd_hires_params_t hires;
     bool circular_x;
     bool circular_y;
-    sd_image_preprocess_params_t image_preprocess;
 } sd_vid_gen_params_t;
 
 typedef struct sd_ctx_t sd_ctx_t;

src/conditioning/conditioner.hpp

@@ -137,7 +137,6 @@ struct ConditionerParams {
     const std::vector<sd::Tensor<float>>* ref_images                    = nullptr;  // for qwen image edit
     const std::vector<MiniMaxH3PresentationItem>* minimax_h3_references = nullptr;
     RefImageParams ref_image_params;
-    bool allow_cache = false;
 };
 
 struct Conditioner {
@@ -1955,10 +1954,6 @@ struct LLMEmbedder : public Conditioner {
     std::shared_ptr<LLM::LLMRunner> llm;
     std::shared_ptr<T5Runner> byt5;
 
-    bool h3_text_cache_valid = false;
-    std::string h3_text_cache_text;
-    SDCondition h3_text_cache;
-
     LLMEmbedder(ggml_backend_t backend,
                 const String2TensorStorage& tensor_storage_map      = {},
                 SDVersion version                                   = VERSION_QWEN_IMAGE,
@@ -2224,10 +2219,7 @@ struct LLMEmbedder : public Conditioner {
                                           false,
                                           deepstack_image_embeds,
                                           image_grids);
-        if (hidden_states.empty()) {
-            LOG_ERROR("LLM prompt encoding failed");
-            return {};
-        }
+        GGML_ASSERT(!hidden_states.empty());
         hidden_states = apply_token_weights(std::move(hidden_states), weights);
         GGML_ASSERT(hidden_states.shape()[1] > prompt_template_encode_start_idx);
 
@@ -2303,25 +2295,6 @@ struct LLMEmbedder : public Conditioner {
 
     SDCondition get_learned_condition(int n_threads,
                                       const ConditionerParams& conditioner_params) override {
-        const bool h3_text_cacheable =
-            sd_version_is_minimax_h3(version) &&
-            conditioner_params.allow_cache &&
-            (conditioner_params.minimax_h3_references == nullptr ||
-             conditioner_params.minimax_h3_references->empty()) &&
-            (conditioner_params.ref_images == nullptr ||
-             conditioner_params.ref_images->empty());
-
-        if (sd_version_is_minimax_h3(version) && !h3_text_cacheable) {
-            h3_text_cache_valid = false;
-        }
-
-        if (h3_text_cacheable &&
-            h3_text_cache_valid &&
-            h3_text_cache_text == conditioner_params.text) {
-            LOG_INFO("H3 conditioning cache hit");
-            return h3_text_cache;
-        }
-
         std::string prompt;
         std::pair<int, int> prompt_attn_range;
         std::vector<std::string> extra_prompts;
@@ -3190,14 +3163,6 @@ struct LLMEmbedder : public Conditioner {
             int64_t tag_count    = static_cast<int64_t>(tags.size());
             result.c_token_types = sd::Tensor<int32_t>({tag_count}, std::move(tags));
         }
-
-        if (h3_text_cacheable) {
-            h3_text_cache_text  = conditioner_params.text;
-            h3_text_cache       = result;
-            h3_text_cache_valid = true;
-            LOG_INFO("H3 conditioning cache stored");
-        }
-
         return result;
     }
 };

src/core/backend_fit.cpp

@@ -478,11 +478,7 @@ namespace sd::backend_fit {
         return true;
     }
 
-    bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling, ggml_status status) {
-        // Execution failures can leave the device unusable; tiling only helps with allocation failures.
-        if (status != GGML_STATUS_ALLOC_FAILED) {
-            return false;
-        }
+    bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling) {
         const char* retry_mode = nullptr;
         if (prefer_temporal_tiling && !tiling_params.temporal_tiling) {
             tiling_params.temporal_tiling = true;
@@ -502,7 +498,7 @@ namespace sd::backend_fit {
             return false;
         }
 
-        LOG_WARN("VAE decode ran out of memory; retrying with %s tiling",
+        LOG_WARN("VAE decode failed (likely out of memory); retrying with %s tiling",
                  retry_mode);
         return true;
     }

src/core/backend_fit.h

@@ -16,8 +16,7 @@ namespace sd::backend_fit {
                               std::string& params_spec);
 
     bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
-                                         bool prefer_temporal_tiling,
-                                         ggml_status status);
+                                         bool prefer_temporal_tiling);
 
 }  // namespace sd::backend_fit
 

src/core/ggml_extend.cpp

@@ -623,7 +623,9 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
                                     bool skip_reshape,
                                     bool flash_attn,
                                     float kv_scale,
-                                    bool sage_attn) {  // avoid overflow
+                                    bool sage_attn,
+                                    bool sol_attn,
+                                    float sol_attn_tau) {  // avoid overflow
     int64_t L_q;
     int64_t L_k;
     int64_t C;
@@ -715,7 +717,23 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
     };
 
 #ifndef SD_USE_UPSTREAM_GGML
-    if (sage_attn && mask == nullptr && d_head > 0 && d_head <= 128) {
+    if (sol_attn && mask == nullptr && d_head == 128 && L_q == L_k && n_head == n_kv_head) {
+        auto q_in = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, q->type == GGML_TYPE_F32 ? q : ggml_cast(ctx, q, GGML_TYPE_F32)), d_head, L_q, n_head, N);
+        auto k_in = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, k->type == GGML_TYPE_F32 ? k : ggml_cast(ctx, k, GGML_TYPE_F32)), d_head, L_k, n_kv_head, N);
+        auto v_in = ggml_ext_cont(ctx, ggml_permute(ctx, v, 0, 2, 1, 3));
+        if (v_in->type != GGML_TYPE_F32) {
+            v_in = ggml_cast(ctx, v_in, GGML_TYPE_F32);
+        }
+        if (kv_scale != 1.0f) {
+            k_in = ggml_ext_scale(ctx, k_in, kv_scale);
+            v_in = ggml_ext_scale(ctx, v_in, kv_scale);
+        }
+        auto out = ggml_sol_attn(ctx, q_in, k_in, v_in, scale / kv_scale, sol_attn_tau);
+        if (ggml_backend_supports_op(backend, out)) {
+            kqv = kv_scale != 1.0f ? ggml_ext_scale(ctx, out, 1.0f / kv_scale) : out;
+        }
+    }
+    if (kqv == nullptr && sage_attn && mask == nullptr && d_head > 0 && d_head <= 128) {
         auto q_in                 = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, q->type == GGML_TYPE_F32 ? q : ggml_cast(ctx, q, GGML_TYPE_F32)), d_head, L_q, n_head, N);
         auto k_in                 = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, k->type == GGML_TYPE_F32 ? k : ggml_cast(ctx, k, GGML_TYPE_F32)), d_head, L_k, n_kv_head, N);
         auto v_in                 = ggml_ext_cont(ctx, ggml_permute(ctx, v, 0, 2, 1, 3));
@@ -744,7 +762,7 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
     }
 #endif
 
-    if (kqv == nullptr && (flash_attn || sage_attn)) {
+    if (kqv == nullptr && (flash_attn || sage_attn || sol_attn)) {
         // LOG_VERBOSE("attention_ext L_q:%d L_k:%d n_head:%d C:%d d_head:%d N:%d", L_q, L_k, n_head, C, d_head, N);
         bool can_use_flash_attn = true;
         if (mask != nullptr) {

src/core/ggml_extend.h

@@ -217,11 +217,13 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
                                     ggml_tensor* k,
                                     ggml_tensor* v,
                                     int64_t n_head,
-                                    ggml_tensor* mask = nullptr,
-                                    bool skip_reshape = false,
-                                    bool flash_attn   = false,
-                                    float kv_scale    = 1.0f,
-                                    bool sage_attn    = false);
+                                    ggml_tensor* mask  = nullptr,
+                                    bool skip_reshape  = false,
+                                    bool flash_attn    = false,
+                                    float kv_scale     = 1.0f,
+                                    bool sage_attn     = false,
+                                    bool sol_attn      = false,
+                                    float sol_attn_tau = 1.0f);
 
 ggml_tensor* ggml_ext_layer_norm(ggml_context* ctx,
                                  ggml_tensor* x,

src/core/ggml_runner.cpp

@@ -25,7 +25,7 @@ ggml_tensor* ggml_ext_attention_ext(GGMLRunnerContext* ctx,
     if (ctx->attn_scale > 0.f) {
         kv_scale = ctx->attn_scale;
     }
-    return ggml_ext_attention_ext(ctx->ggml_ctx, ctx->backend, q, k, v, n_head, mask, skip_reshape, flash_attn, kv_scale, ctx->sage_attn_enabled);
+    return ggml_ext_attention_ext(ctx->ggml_ctx, ctx->backend, q, k, v, n_head, mask, skip_reshape, flash_attn, kv_scale, ctx->sage_attn_enabled, ctx->sol_attn_enabled, ctx->sol_attn_tau);
 }
 
 void GGMLRunner::alloc_params_ctx() {
@@ -164,6 +164,22 @@ ggml_cgraph* GGMLRunner::get_compute_graph(get_graph_cb_t get_graph) {
         }
     }
     prepare_build_in_tensor_after(gf);
+#ifndef SD_USE_UPSTREAM_GGML
+    if (sol_attn_enabled && !sol_attn_graph_logged) {
+        int sol_nodes   = 0;
+        int flash_nodes = 0;
+        for (int i = 0; i < ggml_graph_n_nodes(gf); ++i) {
+            const auto op = ggml_graph_node(gf, i)->op;
+            sol_nodes += op == GGML_OP_SOL_ATTN;
+            flash_nodes += op == GGML_OP_FLASH_ATTN_EXT;
+        }
+        LOG_INFO("Sol-Attn graph: %d Sol-Attn nodes, %d FlashAttention nodes", sol_nodes, flash_nodes);
+        if (sol_nodes == 0) {
+            LOG_WARN("This graph has no attention operations supported by Sol-Attn");
+        }
+        sol_attn_graph_logged = true;
+    }
+#endif
     return gf;
 }
 
@@ -521,6 +537,8 @@ GGMLRunnerContext GGMLRunner::get_context() {
     runner_ctx.backend               = runtime_backend;
     runner_ctx.flash_attn_enabled    = flash_attn_enabled;
     runner_ctx.sage_attn_enabled     = sage_attn_enabled;
+    runner_ctx.sol_attn_enabled      = sol_attn_enabled;
+    runner_ctx.sol_attn_tau          = sol_attn_tau;
     runner_ctx.linear_scale          = linear_scale;
     runner_ctx.attn_scale            = attn_scale;
     runner_ctx.conv2d_direct_enabled = conv2d_direct_enabled;
@@ -590,7 +608,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
                                                      bool auto_runner_end,
                                                      bool no_return,
                                                      const std::function<bool()>& read_outputs) {
-    last_compute_status_ = GGML_STATUS_FAILED;
     if (graph_active_) {
         LOG_ERROR("%s does not support reentrant graph execution", get_desc().c_str());
         return std::nullopt;
@@ -614,9 +631,7 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
         GGMLRunner& runner;
         const bool& success;
         ~GraphEndGuard() {
-            if (!runner.workspace_.segment_end()) {
-                runner.last_compute_status_ = GGML_STATUS_FAILED;
-            }
+            runner.workspace_.segment_end();
             runner.cache_.graph_end(false);
             runner.cut_cache_.clear();
             runner.free_compute_ctx();
@@ -645,7 +660,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
     try {
         output = execute_graph(graph, n_threads, no_return, read_outputs);
     } catch (const std::exception& error) {
-        last_compute_status_ = GGML_STATUS_FAILED;
         LOG_ERROR("%s graph execution failed on %s: %s", get_desc().c_str(),
                   ggml_backend_name(runtime_backend), error.what());
         return std::nullopt;
@@ -653,7 +667,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
     success = output.has_value();
     if (success) {
         cache_.graph_end(true);
-        last_compute_status_ = GGML_STATUS_SUCCESS;
     }
     return output;
 }
@@ -771,7 +784,6 @@ bool GGMLRunner::execute_segment(ggml_cgraph* graph, int n_threads) {
     }
     workspace_.synchronize();
     if (status != GGML_STATUS_SUCCESS) {
-        last_compute_status_ = status;
         LOG_ERROR("%s compute failed: %s", get_desc().c_str(), ggml_status_to_string(status));
         return false;
     }
@@ -824,7 +836,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
     const auto& cached_plan     = resolve_graph_cut_plan(graph);
     const auto full_measurement = measure(graph, cached_plan.compute_buffer_size);
     if (full_measurement.buffers.empty()) {
-        last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
         return std::nullopt;
     }
     auto manager         = residency_manager.lock();
@@ -895,9 +906,7 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
             SegmentGraphBindings& bindings;
             ggml_context* context;
             ~SegmentCleanup() {
-                if (!runner.workspace_.segment_end()) {
-                    runner.last_compute_status_ = GGML_STATUS_FAILED;
-                }
+                runner.workspace_.segment_end();
                 bindings.restore();
                 weights.segment_end();
                 ggml_free(context);
@@ -907,7 +916,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
 
         auto measurement = segmented ? measure(segment_graph, segment.compute_buffer_size) : full_measurement;
         if (!workspace_.prepare(measurement)) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
             return fail_segment("workspace preparation");
         }
         const size_t cut_bytes       = last ? 0 : cut_cache_.estimate_output_bytes(graph, segment);
@@ -922,11 +930,7 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
                 sync_runtime_residency();
                 requests = memory_requests(measurement.buffers, new_cache_bytes);
             }
-            const bool ready = weights.ensure_segment_capacity(index, requests);
-            if (!ready && manager != nullptr) {
-                last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
-            }
-            return ready;
+            return weights.ensure_segment_capacity(index, requests);
         };
         if (!weights.segment_start(index, ensure_capacity)) {
             return fail_segment("weight preparation");
@@ -935,17 +939,12 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
         if (!workspace_.measurement_matches(segment_graph, measurement)) {
             measurement = measure(segment_graph, segment.compute_buffer_size);
         }
-        if (!workspace_.prepare(measurement)) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
-            return fail_segment("workspace preparation");
-        }
-        if (!ensure_capacity()) {
+        if (!workspace_.prepare(measurement) || !ensure_capacity()) {
             return fail_segment("workspace capacity check");
         }
         if (!workspace_.allocate(segment_graph, [&](ggml_backend_sched_t scheduler, ggml_cgraph* current) {
                 pin_multi_device_nodes(scheduler, current);
             })) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
             return fail_segment("workspace allocation");
         }
         for (const auto& size : measurement.buffers) {
@@ -983,7 +982,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
             }
         }
         if (!workspace_.segment_end()) {
-            last_compute_status_ = GGML_STATUS_FAILED;
             return fail_segment("workspace synchronization");
         }
         // Final outputs and their callbacks may still be views of consumed cuts.

src/core/ggml_runner.h

@@ -69,6 +69,8 @@ struct GGMLRunnerContext {
     ggml_context* ggml_ctx                                           = nullptr;
     bool flash_attn_enabled                                          = false;
     bool sage_attn_enabled                                           = false;
+    bool sol_attn_enabled                                            = false;
+    float sol_attn_tau                                               = 1.f;
     float linear_scale                                               = 0.f;
     float attn_scale                                                 = 0.f;
     bool conv2d_direct_enabled                                       = false;
@@ -130,8 +132,7 @@ ggml_tensor* ggml_ext_attention_ext(GGMLRunnerContext* ctx,
 struct GGMLRunner {
 private:
     std::map<ggml_backend_t, size_t> logged_compute_bytes_;
-    size_t logged_segment_count_     = 0;
-    ggml_status last_compute_status_ = GGML_STATUS_SUCCESS;
+    size_t logged_segment_count_ = 0;
 
     sd::ComputeWorkspace::Measurement measure(ggml_cgraph* graph, size_t direct_bytes);
     std::vector<DeviceMemoryRequest> memory_requests(const std::vector<sd::BackendBufferSize>& sizes,
@@ -179,6 +180,9 @@ protected:
 
     bool flash_attn_enabled    = false;
     bool sage_attn_enabled     = false;
+    bool sol_attn_enabled      = false;
+    float sol_attn_tau         = 1.f;
+    bool sol_attn_graph_logged = false;
     float linear_scale         = 0.f;
     float attn_scale           = 0.f;
     bool conv2d_direct_enabled = false;
@@ -336,8 +340,6 @@ public:
                                              bool no_return                            = false,
                                              const std::function<bool()>& read_outputs = {});
 
-    ggml_status last_compute_status() const { return last_compute_status_; }
-
     void set_flash_attention_enabled(bool enabled) {
         flash_attn_enabled = enabled;
     }
@@ -350,6 +352,16 @@ public:
         }
     }
 
+    void set_sol_attention_enabled(bool enabled, float tau) {
+        if (sol_attn_enabled != enabled || sol_attn_tau != tau) {
+            free_cache_ctx_and_buffer();
+            graph_cut_plan_cache_.graph_cut_plans.clear();
+            sol_attn_enabled      = enabled;
+            sol_attn_tau          = tau;
+            sol_attn_graph_logged = false;
+        }
+    }
+
     void set_scale_overrides(float linear_scale, float attn_scale) {
         this->linear_scale = linear_scale;
         this->attn_scale   = attn_scale;

src/detailer.cpp

@@ -970,7 +970,6 @@ bool adetail_image(adetailer_ctx_t* context,
         generation.pm_params           = {};
         generation.pulid_params        = {};
         generation.hires.enabled       = false;
-        generation.image_preprocess    = {};
         if (params.steps > 0) {
             generation.sample_params.sample_steps        = params.steps;
             generation.sample_params.custom_sigmas       = nullptr;

src/model.h

@@ -252,14 +252,6 @@ static inline bool sd_version_is_sensenova_u1(SDVersion version) {
     return version == VERSION_SENSENOVA_U1_5;
 }
 
-static inline bool sd_version_supports_video_generation(SDVersion version) {
-    return version == VERSION_SVD || sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_lingbot_video(version) || sd_version_is_ltxav(version) || sd_version_is_minimax_h3(version);
-}
-
-static inline bool sd_version_supports_image_generation(SDVersion version) {
-    return !sd_version_supports_video_generation(version);
-}
-
 static inline bool sd_version_uses_flux_vae(SDVersion version) {
     if (sd_version_is_flux(version) || sd_version_is_z_image(version) || sd_version_is_boogu_image(version) || sd_version_is_longcat(version)) {
         return true;

src/model_manager.cpp

@@ -1613,8 +1613,7 @@ void ModelManager::remove_runtime_owner(uintptr_t owner_id) {
 
 ModelManager::CapacityCheck ModelManager::check_capacity(
     const DeviceMemoryRequest& request,
-    const std::vector<TensorState*>& states,
-    bool log_details) const {
+    const std::vector<TensorState*>& states) const {
     CapacityCheck result;
     if (request.compute_backend == nullptr || sd_backend_is_cpu(request.compute_backend)) {
         return result;
@@ -1632,23 +1631,16 @@ ModelManager::CapacityCheck ModelManager::check_capacity(
         }
         size_t free_bytes = 0, total_bytes = 0;
         ggml_backend_dev_memory(device, &free_bytes, &total_bytes);
-        const size_t weights_resident = compute_backend_resident_bytes(backend);
-        const size_t other_runtime    = other_runtime_resident_bytes(request.owner_id, backend);
-        const size_t resident         = add(weights_resident, add(other_runtime, request.runtime_resident_bytes));
-        if (log_details) {
-            LOG_WARN("model manager memory on %s: reported free %.2f MB / total %.2f MB, tracked weights %.2f MB / other runtime %.2f MB / current runtime %.2f MB",
-                        ggml_backend_name(backend),
-                        free_bytes / (1024.0 * 1024.0), total_bytes / (1024.0 * 1024.0),
-                        weights_resident / (1024.0 * 1024.0), other_runtime / (1024.0 * 1024.0),
-                        request.runtime_resident_bytes / (1024.0 * 1024.0));
-        }
         if (free_bytes == 0 && total_bytes == 0) {
             return SIZE_MAX;
         }
         // Vulkan's heap budget subtraction can underflow when usage exceeds the budget.
-        if (total_bytes > 0 && free_bytes > total_bytes && sd_backend_is(backend, "Vulkan")) {
+        if (total_bytes > 0 && free_bytes > total_bytes) {
             return size_t{0};
         }
+        const size_t resident = add(compute_backend_resident_bytes(backend),
+                                       add(other_runtime_resident_bytes(request.owner_id, backend),
+                                           request.runtime_resident_bytes));
         if (total_bytes > 0) {
             free_bytes = std::min(free_bytes, resident < total_bytes ? total_bytes - resident : 0);
         }
@@ -1794,7 +1786,7 @@ bool ModelManager::ensure_compute_backend_capacity(
         }
     }
 
-    const auto capacity                = check_capacity(request, required_states, true);
+    const auto capacity                = check_capacity(request, required_states);
     const std::string available_device = capacity.available_device_bytes == SIZE_MAX
                                              ? "unknown"
                                              : sd_format("%.2f MB", capacity.available_device_bytes / (1024.0 * 1024.0));

src/model_manager.h

@@ -157,8 +157,7 @@ private:
         }
     };
     CapacityCheck check_capacity(const DeviceMemoryRequest& request,
-                                 const std::vector<TensorState*>& states,
-                                 bool log_details = false) const;
+                                 const std::vector<TensorState*>& states) const;
 
     ggml_backend_buffer_type_t params_buffer_type_for(const TensorState& state) const;
     ggml_backend_buffer_type_t split_buffer_type_for(const TensorState& state) const;

src/pipeline/diffusion_engine.cpp

@@ -899,7 +899,60 @@ bool StableDiffusionGGML::set_sage_attention_enabled(bool enabled) {
     return true;
 }
 
+bool StableDiffusionGGML::set_sol_attention_enabled(bool enabled, float tau) {
+    if (!diffusion_model || !std::isfinite(tau)) {
+        LOG_ERROR("Sol-Attn requires a diffusion model and finite tau");
+        return false;
+    }
+    if (enabled) {
+        if (config_->params.sage_attn) {
+            LOG_ERROR("Sol-Attn and SageAttention cannot be enabled together");
+            return false;
+        }
+#ifndef SD_USE_UPSTREAM_GGML
+        auto* ctx = ggml_init({4 * ggml_tensor_overhead(), nullptr, true});
+        if (!ctx) {
+            return false;
+        }
+        auto* q        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* k        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* v        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* op       = ggml_sol_attn(ctx, q, k, v, 1.f / sqrtf(128.f), tau);
+        bool supported = true;
+        for (auto backend : backend_manager.runtime_backends(SDBackendModule::DIFFUSION)) {
+            if (!ggml_backend_supports_op(backend, op)) {
+                LOG_ERROR("Sol-Attn is unavailable on %s; it requires patched GGML, CUDA 12.0 or newer, and SM80 or newer kernels", ggml_backend_name(backend));
+                supported = false;
+            }
+        }
+        ggml_free(ctx);
+        if (!supported) {
+            return false;
+        }
+#else
+        LOG_ERROR("Sol-Attn requires -DSD_USE_UPSTREAM_GGML=OFF and a CUDA backend");
+        return false;
+#endif
+    }
+    diffusion_model->set_sol_attention_enabled(enabled, tau);
+    if (high_noise_diffusion_model) {
+        high_noise_diffusion_model->set_sol_attention_enabled(enabled, tau);
+    }
+    if (enabled) {
+        LOG_INFO("Using Sol-Attn (tau=%g, diagonal threshold) in diffusion; unsupported attention uses flash/default attention", tau);
+    }
+    return true;
+}
+
 bool StableDiffusionGGML::init(const sd_ctx_params_t* sd_ctx_params) {
+    if (sd_ctx_params->sol_attn && sd_ctx_params->sage_attn) {
+        LOG_ERROR("Sol-Attn and SageAttention cannot be enabled together");
+        return false;
+    }
+    if (!std::isfinite(sd_ctx_params->sol_attn_tau)) {
+        LOG_ERROR("Sol-Attn tau must be finite");
+        return false;
+    }
 #ifdef SD_USE_UPSTREAM_GGML
     LOG_WARN(
         "Using upstream GGML: INT8 tensorwise/convrot is disabled and FP8 weights are "
@@ -1180,6 +1233,9 @@ bool StableDiffusionGGML::validate_and_load_runners() {
     if (sd_ctx_params->sage_attn && !set_sage_attention_enabled(true)) {
         return false;
     }
+    if (sd_ctx_params->sol_attn && !set_sol_attention_enabled(true, sd_ctx_params->sol_attn_tau)) {
+        return false;
+    }
     LOG_VERBOSE("validating model metadata");
 
     std::set<std::string> ignore_tensors;
@@ -1762,8 +1818,6 @@ bool StableDiffusionGGML::apply_loras(const sd_lora_t* loras, uint32_t lora_coun
         extension->collect_loras(all_loras);
     }
 
-    conditioning_cache_allowed_ = all_loras.empty();
-
     int64_t t0 = ggml_time_ms();
     end_runners();
     clear_lora_adapters();
@@ -2768,8 +2822,7 @@ sd::Tensor<float> StableDiffusionGGML::decode_first_stage(const sd::Tensor<float
     auto decoded                      = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
     const bool prefer_temporal_tiling = decode_video && first_stage_model->can_temporal_tile_decode();
     while (decoded.empty() &&
-           sd::backend_fit::prepare_vae_decode_retry_tiling(vae_tiling_params, prefer_temporal_tiling,
-                                                            first_stage_model->last_compute_status())) {
+           sd::backend_fit::prepare_vae_decode_retry_tiling(vae_tiling_params, prefer_temporal_tiling)) {
         decoded = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
     }
     return decoded;

src/pipeline/diffusion_engine.h

@@ -178,7 +178,6 @@ public:
     std::recursive_mutex execution_mutex;
     std::unique_ptr<ModelConfig> config_;
     RunnerState runner_state_;
-    bool conditioning_cache_allowed_ = false;
     bool executing_ = false;
 
     std::shared_ptr<Denoiser> denoiser;
@@ -314,6 +313,7 @@ public:
 
     bool init(const sd_ctx_params_t* sd_ctx_params);
     bool set_sage_attention_enabled(bool enabled);
+    bool set_sol_attention_enabled(bool enabled, float tau);
 
     bool uses_tae() const;
 

src/pipeline/image.cpp

@@ -10,7 +10,6 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
-#include "runtime/image_preprocess.h"
 #include "upscaler.h"
 
 namespace sd::pipeline {
@@ -790,9 +789,15 @@ namespace sd::pipeline {
             return false;
         }
 
-        if (!sd_version_supports_image_generation(sd->version)) {
-            LOG_ERROR("%s cannot be run with generate_image(); use generate_video() or --mode vid_gen in the CLI",
-                      model_version_to_str[sd->version]);
+        // MiniMax-H3 is video-only. Its denoiser always splits the packed latent into a video and an
+        // audio half, and only generate_video ever computes the audio length, so reaching this
+        // function with an H3 checkpoint is guaranteed to die on
+        // GGML_ASSERT(!audio_input_cache.empty()) with a core dump, after the several minutes it
+        // takes to load the weights, and with nothing in the output pointing at the missing --mode.
+        // (The AnimateDiff path below routes vid_gen back through here, but that is SD1.5 plus a
+        // motion module, never H3.)
+        if (sd_version_is_minimax_h3(sd->version)) {
+            LOG_ERROR("MiniMax-H3 is a video model and cannot be run in img_gen mode; use --mode vid_gen");
             return false;
         }
 
@@ -801,12 +806,6 @@ namespace sd::pipeline {
         int64_t t0            = ggml_time_ms();
         sd->vae_tiling_params = sd_img_gen_params->vae_tiling_params;
         GenerationRequest request(sd, sd_img_gen_params);
-        sd::ImagePreprocessor preprocessing(sd_img_gen_params->image_preprocess.rules);
-        sd_img_gen_params_t processed_params = *sd_img_gen_params;
-        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
-            return false;
-        sd_img_gen_params = &processed_params;
-        request.pm_params = processed_params.pm_params;
         LOG_INFO("generate_image %dx%d", request.width, request.height);
 
         sd->rng->manual_seed(request.seed);

src/pipeline/video.cpp

@@ -15,7 +15,6 @@
 #include "model/vae/vae.hpp"
 #include "request.h"
 #include "runtime/denoiser.hpp"
-#include "runtime/image_preprocess.h"
 
 namespace sd::pipeline {
 
@@ -471,15 +470,11 @@ namespace sd::pipeline {
         sd::Tensor<float> end_image;
 
         if (sd_vid_gen_params->init_image.data) {
-            start_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height),
-                sd->get_image_channels());
+            start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height);
         }
 
         if (sd_vid_gen_params->end_image.data) {
-            end_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height),
-                sd->get_image_channels());
+            end_image = sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height);
         }
 
         if (sd_version_is_minimax_h3(sd->version)) {
@@ -1157,10 +1152,6 @@ namespace sd::pipeline {
         condition_params.zero_out_masked       = true;
         condition_params.ref_images            = &latents.ref_images;
         condition_params.minimax_h3_references = &latents.minimax_presentation_refs;
-        condition_params.allow_cache =
-            sd_version_is_minimax_h3(sd->version) &&
-            sd->conditioning_cache_allowed_ &&
-            !request.use_uncond;
         if (sd_version_is_lingbot_video(sd->version) || sd_version_is_minimax_h3(sd->version)) {
             condition_params.ref_image_params.vlm_resize_mode = RefImageResizeMode::AREA;
         }
@@ -1425,9 +1416,7 @@ namespace sd::pipeline {
         sd::Tensor<float> video_mask = make_ltxav_video_denoise_mask(video_latent, 1.f);
 
         if (sd_vid_gen_params->init_image.data != nullptr) {
-            sd::Tensor<float> start_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height),
-                sd->get_image_channels());
+            sd::Tensor<float> start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height);
             if (!apply_ltxav_condition_image_by_latent_index(sd,
                                                              start_image,
                                                              &video_latent,
@@ -1440,9 +1429,7 @@ namespace sd::pipeline {
         }
 
         if (sd_vid_gen_params->end_image.data != nullptr) {
-            sd::Tensor<float> end_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height),
-                sd->get_image_channels());
+            sd::Tensor<float> end_image        = sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height);
             sd::Tensor<float> end_image_latent = encode_ltxav_condition_image(sd, end_image, "end");
             if (end_image_latent.empty()) {
                 return false;
@@ -1531,7 +1518,6 @@ namespace sd::pipeline {
         img_gen_params.qwen_image_layers = 0;
         img_gen_params.circular_x        = sd_vid_gen_params->circular_x;
         img_gen_params.circular_y        = sd_vid_gen_params->circular_y;
-        img_gen_params.image_preprocess  = sd_vid_gen_params->image_preprocess;
 
         sd->animatediff_num_frames = n_frames;
         bool ok                    = generate_image(sd, &img_gen_params, frames_out, num_frames_out);
@@ -1562,11 +1548,6 @@ namespace sd::pipeline {
         sd->vae_tiling_params = sd_vid_gen_params->vae_tiling_params;
         sd->apply_circular_axes(sd_vid_gen_params->circular_x, sd_vid_gen_params->circular_y);
         GenerationRequest request(sd, sd_vid_gen_params);
-        sd::ImagePreprocessor preprocessing(sd_vid_gen_params->image_preprocess.rules);
-        sd_vid_gen_params_t processed_params = *sd_vid_gen_params;
-        if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
-            return false;
-        sd_vid_gen_params = &processed_params;
         if (fps_out != nullptr) {
             *fps_out = request.fps;
         }

src/runtime/image_preprocess.cpp

@@ -1,447 +0,0 @@
-#include "image_preprocess.h"
-#include "core/util.h"
-
-#include <climits>
-#include <set>
-
-namespace sd {
-
-    static constexpr std::pair<const char*, ImageTarget> image_targets[] = {
-        {"init", ImageTarget::Init},
-        {"end", ImageTarget::End},
-        {"mask", ImageTarget::Mask},
-        {"control", ImageTarget::Control},
-        {"ref", ImageTarget::Ref},
-        {"ip-adapter", ImageTarget::IPAdapter},
-        {"id", ImageTarget::ID},
-        {"control-frame", ImageTarget::ControlFrame},
-    };
-
-    static constexpr std::pair<const char*, ImageResizeMode> image_resize_modes[] = {
-        {"auto", ImageResizeMode::Auto},
-        {"none", ImageResizeMode::None},
-        {"stretch", ImageResizeMode::Stretch},
-        {"crop", ImageResizeMode::Crop},
-        {"crop-resize", ImageResizeMode::CropResize},
-        {"fit-pad", ImageResizeMode::FitPad},
-    };
-
-    template <typename T, size_t N>
-    static bool parse_enum(const std::string& text, const std::pair<const char*, T> (&names)[N], T& value) {
-        for (const auto& entry : names) {
-            if (text == entry.first) {
-                value = entry.second;
-                return true;
-            }
-        }
-        return false;
-    }
-
-    template <typename T, size_t N>
-    static const char* enum_name(T value, const std::pair<const char*, T> (&names)[N]) {
-        for (const auto& entry : names) {
-            if (value == entry.second)
-                return entry.first;
-        }
-        return "unknown";
-    }
-
-    template <typename T>
-    static bool one_of(T value, std::initializer_list<T> choices) {
-        return std::find(choices.begin(), choices.end(), value) != choices.end();
-    }
-
-    static bool one_of(const std::string& value, std::initializer_list<const char*> choices) {
-        for (const char* choice : choices) {
-            if (value == choice)
-                return true;
-        }
-        return false;
-    }
-
-    static ImageResizeMode resolve_mode(const std::map<std::string, std::string>& options, ImageResizeMode default_mode) {
-        auto it              = options.find("mode");
-        ImageResizeMode mode = ImageResizeMode::Auto;
-        if (it != options.end())
-            parse_enum(it->second, image_resize_modes, mode);
-        if (mode != ImageResizeMode::Auto)
-            return mode;
-        return options.count("width") && default_mode == ImageResizeMode::None ? ImageResizeMode::Stretch : default_mode;
-    }
-
-    bool ImagePreprocessor::fail(const std::string& message) const {
-        LOG_ERROR("image preprocessing: %s", message.c_str());
-        valid_ = false;
-        return false;
-    }
-
-    ImagePreprocessor::ImagePreprocessor(const char* text) {
-        if (text == nullptr || trim(text).empty())
-            return;
-        for (const auto& part : split_string(text, ';')) {
-            ImagePreprocessRule rule;
-            std::set<std::string> keys;
-            if (trim(part).empty()) {
-                fail("empty rule");
-                return;
-            }
-            for (const auto& entry : split_string(part, ',')) {
-                size_t equal = entry.find('=');
-                if (equal == std::string::npos) {
-                    fail("expected key=value: " + entry);
-                    return;
-                }
-                std::string key   = trim(entry.substr(0, equal));
-                std::string value = trim(entry.substr(equal + 1));
-                bool ok           = !value.empty() && keys.insert(key).second;
-                int number        = 0;
-                if (key == "target") {
-                    ok &= parse_enum(value, image_targets, rule.target);
-                } else if (key == "index") {
-                    ok &= parse_strict_int(value, rule.index) && rule.index >= 0;
-                } else {
-                    if (key == "mode") {
-                        ImageResizeMode mode;
-                        ok &= parse_enum(value, image_resize_modes, mode);
-                    } else if (key == "filter") {
-                        ok &= one_of(value, {"auto", "nearest", "nearest-exact", "bilinear", "bicubic", "lanczos"});
-                    } else if (key == "antialias") {
-                        ok &= one_of(value, {"auto", "true", "false"});
-                    } else if (key == "canny") {
-                        ok &= one_of(value, {"true", "false"});
-                    } else if (key == "anchor") {
-                        ok &= one_of(value, {"center", "top", "bottom", "left", "right"});
-                    } else if (key == "width" || key == "height") {
-                        ok &= parse_strict_int(value, number) && number > 0;
-                    } else if (key == "pad_color") {
-                        ok &= value.size() == 7 || value.size() == 9;
-                        ok &= !value.empty() && value[0] == '#';
-                        for (size_t i = 1; i < value.size(); ++i)
-                            ok &= std::isxdigit(static_cast<unsigned char>(value[i])) != 0;
-                    } else {
-                        ok = false;
-                    }
-                    rule.options[key] = value;
-                }
-                if (!ok) {
-                    fail("invalid or duplicate option: " + entry);
-                    return;
-                }
-            }
-            if (!keys.count("target") || rule.options.empty() ||
-                rule.options.count("width") != rule.options.count("height") ||
-                (rule.index >= 0 && !one_of(rule.target, {ImageTarget::Ref, ImageTarget::ID, ImageTarget::ControlFrame}))) {
-                fail("invalid target, index, or incomplete dimensions: " + part);
-                return;
-            }
-            rules_.push_back(std::move(rule));
-        }
-        for (const auto& rule : rules_) {
-            const auto options = resolve_options(rule.target, std::max(0, rule.index));
-            if (options.count("antialias") && options.at("antialias") == "true" && options.count("filter") &&
-                one_of(options.at("filter"), {"nearest", "nearest-exact"})) {
-                fail("antialias requires bilinear, bicubic, or lanczos");
-                return;
-            }
-        }
-    }
-
-    std::map<std::string, std::string> ImagePreprocessor::resolve_options(ImageTarget target, int index) const {
-        std::map<std::string, std::string> options;
-        for (int specificity = 0; specificity < 2; ++specificity) {
-            for (const auto& rule : rules_) {
-                if (rule.target == target &&
-                    rule.index == (specificity == 0 ? -1 : index)) {
-                    for (const auto& entry : rule.options)
-                        options[entry.first] = entry.second;
-                }
-            }
-        }
-        return options;
-    }
-
-    bool ImagePreprocessor::validate_inputs(const sd_img_gen_params_t& params) const {
-        const std::map<ImageTarget, int> counts = {
-            {ImageTarget::Init, params.init_image.data != nullptr},
-            {ImageTarget::Mask, params.mask_image.data != nullptr},
-            {ImageTarget::Control, params.control_image.data != nullptr},
-            {ImageTarget::IPAdapter, params.ip_adapter_image.data != nullptr},
-            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
-            {ImageTarget::ID, params.pm_params.id_images != nullptr ? params.pm_params.id_images_count : 0},
-        };
-        for (const auto& rule : rules_) {
-            auto it   = counts.find(rule.target);
-            int count = it == counts.end() ? 0 : it->second;
-            if (count <= 0 || rule.index >= count) {
-                return fail(std::string("rule targets an unavailable image: ") + enum_name(rule.target, image_targets));
-            }
-        }
-        return valid_;
-    }
-
-    bool ImagePreprocessor::validate_inputs(const sd_vid_gen_params_t& params) const {
-        const std::map<ImageTarget, int> counts = {
-            {ImageTarget::Init, params.init_image.data != nullptr},
-            {ImageTarget::End, params.end_image.data != nullptr},
-            {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
-            {ImageTarget::ControlFrame, params.control_frames != nullptr ? params.control_frames_size : 0},
-        };
-        for (const auto& rule : rules_) {
-            auto it   = counts.find(rule.target);
-            int count = it == counts.end() ? 0 : it->second;
-            if (count <= 0 || rule.index >= count)
-                return fail(std::string("rule targets an unavailable video input: ") + enum_name(rule.target, image_targets));
-        }
-        return valid_;
-    }
-
-    static int anchor_offset(int remaining, const std::string& anchor, bool horizontal) {
-        if (anchor == (horizontal ? "left" : "top"))
-            return 0;
-        if (anchor == (horizontal ? "right" : "bottom"))
-            return remaining;
-        return remaining / 2;
-    }
-
-    Tensor<float> ImagePreprocessor::apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform p, const std::string& label, ops::InterpolateMode default_filter) const {
-        auto value = [&](const char* key, const char* fallback) {
-            auto it = options.find(key);
-            return it == options.end() ? std::string(fallback) : it->second;
-        };
-        std::string filter        = value("filter", "auto");
-        ops::InterpolateMode mode = default_filter;
-        if (filter == "nearest")
-            mode = ops::InterpolateMode::Nearest;
-        if (filter == "nearest-exact")
-            mode = ops::InterpolateMode::NearestExact;
-        if (filter == "bilinear")
-            mode = ops::InterpolateMode::Bilinear;
-        if (filter == "bicubic")
-            mode = ops::InterpolateMode::Bicubic;
-        if (filter == "lanczos")
-            mode = ops::InterpolateMode::Lanczos;
-        bool filtered  = ops::is_2d_filter_interpolate_mode(mode);
-        bool antialias = value("antialias", "auto") == "true" ||
-                         (value("antialias", "auto") == "auto" && filtered &&
-                          (p.resize_width < p.crop_width || p.resize_height < p.crop_height));
-        if (antialias && !filtered) {
-            fail(label + ": antialias requires bilinear, bicubic, or lanczos");
-            return {};
-        }
-        auto cropped = ops::slice(ops::slice(image, 0, p.x, p.x + p.crop_width), 1, p.y, p.y + p.crop_height);
-        int channels = static_cast<int>(image.shape()[2]);
-        bool resize  = p.resize_width != p.crop_width || p.resize_height != p.crop_height;
-        if (resize && channels == 4 && filtered) {
-            for (int64_t i = 0, pixels = cropped.shape()[0] * cropped.shape()[1]; i < pixels; ++i) {
-                for (int c = 0; c < 3; ++c)
-                    cropped[i + c * pixels] *= cropped[i + 3 * pixels];
-            }
-        }
-        auto resized = ops::interpolate(cropped, {p.resize_width, p.resize_height, channels, 1}, mode, false, antialias);
-        if (resize && channels == 4 && filtered) {
-            for (int64_t i = 0, pixels = resized.shape()[0] * resized.shape()[1]; i < pixels; ++i) {
-                float alpha = std::clamp(resized[i + 3 * pixels], 0.f, 1.f);
-                for (int c = 0; c < 3; ++c)
-                    resized[i + c * pixels] = alpha > 1e-6f ? resized[i + c * pixels] / alpha : 0.f;
-            }
-        }
-        resized = ops::clamp(resized, 0.f, 1.f);
-        Tensor<float> output({p.width, p.height, channels, 1});
-        std::string color = value("pad_color", "#000000ff");
-        if (color.size() == 7)
-            color += "ff";
-        uint8_t rgba[4];
-        for (int c = 0; c < 4; ++c)
-            rgba[c] = static_cast<uint8_t>(std::strtoul(color.substr(1 + c * 2, 2).c_str(), nullptr, 16));
-        for (int c = 0; c < channels; ++c) {
-            float fill = rgba[channels == 1 ? 0 : c] / 255.f;
-            for (int y = 0; y < p.height; ++y) {
-                for (int x = 0; x < p.width; ++x) {
-                    output.index(x, y, c, 0) = x >= p.pad_x && x < p.pad_x + p.resize_width && y >= p.pad_y && y < p.pad_y + p.resize_height
-                                                   ? resized.index(x - p.pad_x, y - p.pad_y, c, 0)
-                                                   : fill;
-                }
-            }
-        }
-        LOG_INFO("preprocess %s: %dx%d crop=(%d,%d,%d,%d) resize=%dx%d pad=(%d,%d) output=%dx%d filter=%s(%d) antialias=%s",
-                 label.c_str(), p.source_width, p.source_height, p.x, p.y, p.crop_width, p.crop_height,
-                 p.resize_width, p.resize_height, p.pad_x, p.pad_y, p.width, p.height, filter.c_str(), static_cast<int>(mode), BOOL_STR(antialias));
-        return output;
-    }
-
-    Tensor<float> ImagePreprocessor::apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode, ops::InterpolateMode default_filter, ImageTransform* plan_out) const {
-        if (!valid_ || image.empty())
-            return {};
-        const std::string label = std::string(enum_name(target, image_targets)) + "[" + std::to_string(index) + "]";
-        auto options            = resolve_options(target, index);
-        if (image.dim() != 4 || image.shape()[3] != 1 || image.shape()[2] < 1 || image.shape()[2] > 4) {
-            fail(label + ": expected one image with 1 to 4 channels");
-            return {};
-        }
-        ImageTransform p;
-        p.source_width = p.crop_width = static_cast<int>(image.shape()[0]);
-        p.source_height = p.crop_height = static_cast<int>(image.shape()[1]);
-        int target_width                = width > 0 ? width : p.source_width;
-        int target_height               = height > 0 ? height : p.source_height;
-        if (options.count("width")) {
-            parse_strict_int(options.at("width"), target_width);
-            parse_strict_int(options.at("height"), target_height);
-        }
-        ImageResizeMode mode = resolve_mode(options, default_mode);
-        std::string anchor   = options.count("anchor") ? options.at("anchor") : "center";
-        p.width = p.resize_width = target_width;
-        p.height = p.resize_height = target_height;
-        if (mode == ImageResizeMode::None) {
-            if (options.count("width") && (target_width != p.source_width || target_height != p.source_height)) {
-                fail(label + ": mode=none conflicts with requested dimensions");
-                return {};
-            }
-            p.width = p.resize_width = p.source_width;
-            p.height = p.resize_height = p.source_height;
-        } else if (mode == ImageResizeMode::Crop || mode == ImageResizeMode::CropResize) {
-            if (mode == ImageResizeMode::Crop) {
-                p.crop_width  = target_width;
-                p.crop_height = target_height;
-            } else if (int64_t(p.source_width) * target_height > int64_t(p.source_height) * target_width) {
-                p.crop_width = std::max(1, static_cast<int>(int64_t(p.source_height) * target_width / target_height));
-            } else {
-                p.crop_height = std::max(1, static_cast<int>(int64_t(p.source_width) * target_height / target_width));
-            }
-            if (p.crop_width > p.source_width || p.crop_height > p.source_height) {
-                fail(label + ": crop exceeds source dimensions");
-                return {};
-            }
-            p.x = anchor_offset(p.source_width - p.crop_width, anchor, true);
-            p.y = anchor_offset(p.source_height - p.crop_height, anchor, false);
-        } else if (mode == ImageResizeMode::FitPad) {
-            double scale    = std::min(double(target_width) / p.source_width, double(target_height) / p.source_height);
-            p.resize_width  = std::max(1, std::min(target_width, static_cast<int>(std::round(p.source_width * scale))));
-            p.resize_height = std::max(1, std::min(target_height, static_cast<int>(std::round(p.source_height * scale))));
-            p.pad_x         = anchor_offset(target_width - p.resize_width, anchor, true);
-            p.pad_y         = anchor_offset(target_height - p.resize_height, anchor, false);
-        }
-        if (p.width <= 0 || p.height <= 0) {
-            fail(label + ": invalid output dimensions");
-            return {};
-        }
-        uint64_t max_pixels = std::min<uint64_t>(INT64_MAX, SIZE_MAX / sizeof(float)) / static_cast<uint64_t>(image.shape()[2]);
-        if (uint64_t(p.width) * p.height > max_pixels || uint64_t(p.resize_width) * p.resize_height > max_pixels) {
-            fail(label + ": image allocation size overflows");
-            return {};
-        }
-        if (plan_out != nullptr)
-            *plan_out = p;
-        return apply_transform(image, options, p, label, default_filter);
-    }
-
-    Tensor<float> ImagePreprocessor::preprocess_input(sd_image_t image, ImageTarget target, int index, int width, int height) {
-        if (image.data == nullptr || image.width == 0 || image.height == 0 || image.width > INT_MAX || image.height > INT_MAX || image.channel < 1 || image.channel > 4) {
-            fail(std::string(enum_name(target, image_targets)) + ": invalid input image");
-            return {};
-        }
-        auto tensor = sd_image_to_tensor(image);
-        if (target == ImageTarget::Mask && has_init_transform_) {
-            auto options = resolve_options(target, index);
-            if (image.width != init_transform_.source_width || image.height != init_transform_.source_height) {
-                fail("mask and init source dimensions must match");
-                return {};
-            }
-            bool geometry_override = options.count("width") || options.count("anchor") ||
-                                     (options.count("mode") && options.at("mode") != "auto");
-            if (geometry_override) {
-                ImageTransform p;
-                auto init_options            = resolve_options(ImageTarget::Init, 0);
-                ImageResizeMode default_mode = resolve_mode(init_options, ImageResizeMode::CropResize);
-                auto result                  = apply_geometry(tensor, target, index, init_transform_.width, init_transform_.height, default_mode, ops::InterpolateMode::NearestExact, &p);
-                if (result.empty())
-                    return {};
-                const auto& q = init_transform_;
-                if (p.x != q.x || p.y != q.y || p.crop_width != q.crop_width || p.crop_height != q.crop_height ||
-                    p.resize_width != q.resize_width || p.resize_height != q.resize_height || p.pad_x != q.pad_x || p.pad_y != q.pad_y || p.width != q.width || p.height != q.height) {
-                    fail("mask geometry conflicts with init; configure geometry on init and filter on mask");
-                    return {};
-                }
-                return result;
-            }
-            return apply_transform(tensor, options, init_transform_, "mask[0]", ops::InterpolateMode::NearestExact);
-        }
-        auto result = apply_geometry(tensor, target, index, width, height, width > 0 ? ImageResizeMode::CropResize : ImageResizeMode::None,
-                                     target == ImageTarget::Mask ? ops::InterpolateMode::NearestExact : ops::InterpolateMode::Nearest,
-                                     target == ImageTarget::Init ? &init_transform_ : nullptr);
-        if (target == ImageTarget::Init)
-            has_init_transform_ = !result.empty();
-        return result;
-    }
-
-    ImagePreprocessor::~ImagePreprocessor() {
-        for (const auto& image : owned_images_)
-            std::free(image.data);
-    }
-
-    bool ImagePreprocessor::prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height) {
-        if (image.data == nullptr)
-            return true;
-        auto options = resolve_options(target, index);
-        bool canny   = options.count("canny") && options.at("canny") == "true";
-        auto tensor  = preprocess_input(image, target, index, width, height);
-        if (tensor.empty())
-            return false;
-        auto output = tensor_to_sd_image(tensor);
-        if (output.data == nullptr)
-            return fail("could not allocate input preprocessing buffer");
-        owned_images_.push_back(output);
-        if (canny && !preprocess_canny(output, 0.08f, 0.08f, 0.8f, 1.f, false))
-            return fail("Canny preprocessing failed");
-        image = output;
-        return true;
-    }
-
-    bool ImagePreprocessor::prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width, int height) {
-        if (count < 0 || (count > 0 && images == nullptr))
-            return fail(std::string("invalid image array: ") + enum_name(target, image_targets));
-        if (count == 0)
-            return true;
-        storage.assign(images, images + count);
-        for (int i = 0; i < count; ++i) {
-            if (storage[i].data == nullptr)
-                return fail(std::string("empty image in array: ") + enum_name(target, image_targets));
-            if (!prepare_image(storage[i], target, i, width, height))
-                return false;
-        }
-        images = storage.data();
-        return true;
-    }
-
-    bool ImagePreprocessor::prepare_inputs(sd_img_gen_params_t& params, int width, int height) {
-        if (prepared_)
-            return fail("inputs have already been prepared");
-        prepared_ = true;
-        if (!valid_ || !validate_inputs(params))
-            return false;
-        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
-            !prepare_image(params.mask_image, ImageTarget::Mask, 0, width, height) ||
-            !prepare_image(params.control_image, ImageTarget::Control, 0, width, height) ||
-            !prepare_image(params.ip_adapter_image, ImageTarget::IPAdapter, 0, -1, -1) ||
-            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
-            !prepare_array(params.pm_params.id_images, params.pm_params.id_images_count, ImageTarget::ID, id_images_))
-            return false;
-        params.image_preprocess = {};
-        return true;
-    }
-
-    bool ImagePreprocessor::prepare_inputs(sd_vid_gen_params_t& params, int width, int height) {
-        if (prepared_)
-            return fail("inputs have already been prepared");
-        prepared_ = true;
-        if (!valid_ || !validate_inputs(params))
-            return false;
-        if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
-            !prepare_image(params.end_image, ImageTarget::End, 0, width, height) ||
-            !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
-            !prepare_array(params.control_frames, params.control_frames_size, ImageTarget::ControlFrame, control_frames_, width, height))
-            return false;
-        params.image_preprocess = {};
-        return true;
-    }
-
-}  // namespace sd

src/runtime/image_preprocess.h

@@ -1,88 +0,0 @@
-#ifndef __SD_RUNTIME_IMAGE_PREPROCESS_H__
-#define __SD_RUNTIME_IMAGE_PREPROCESS_H__
-
-#include <map>
-#include <string>
-#include <vector>
-
-#include "core/tensor.hpp"
-#include "stable-diffusion.h"
-
-namespace sd {
-
-    enum class ImageTarget {
-        Init,
-        End,
-        Mask,
-        Control,
-        Ref,
-        IPAdapter,
-        ID,
-        ControlFrame,
-    };
-
-    enum class ImageResizeMode {
-        Auto,
-        None,
-        Stretch,
-        Crop,
-        CropResize,
-        FitPad,
-    };
-
-    struct ImageTransform {
-        int source_width  = 0;
-        int source_height = 0;
-        int x             = 0;
-        int y             = 0;
-        int crop_width    = 0;
-        int crop_height   = 0;
-        int resize_width  = 0;
-        int resize_height = 0;
-        int width         = 0;
-        int height        = 0;
-        int pad_x         = 0;
-        int pad_y         = 0;
-    };
-
-    struct ImagePreprocessRule {
-        ImageTarget target = ImageTarget::Init;
-        int index          = -1;
-        std::map<std::string, std::string> options;
-    };
-
-    class ImagePreprocessor {
-        std::vector<ImagePreprocessRule> rules_;
-        mutable bool valid_ = true;
-        ImageTransform init_transform_;
-        bool has_init_transform_ = false;
-        bool prepared_           = false;
-        std::vector<sd_image_t> owned_images_;
-        std::vector<sd_image_t> ref_images_;
-        std::vector<sd_image_t> id_images_;
-        std::vector<sd_image_t> control_frames_;
-
-        bool fail(const std::string& message) const;
-        std::map<std::string, std::string> resolve_options(ImageTarget target, int index) const;
-        Tensor<float> apply_transform(const Tensor<float>& image, const std::map<std::string, std::string>& options, ImageTransform plan, const std::string& label, ops::InterpolateMode default_filter) const;
-
-        bool prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height);
-        bool prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector<sd_image_t>& storage, int width = -1, int height = -1);
-
-    public:
-        explicit ImagePreprocessor(const char* rules = nullptr);
-        ~ImagePreprocessor();
-        ImagePreprocessor(const ImagePreprocessor&)            = delete;
-        ImagePreprocessor& operator=(const ImagePreprocessor&) = delete;
-        bool prepare_inputs(sd_img_gen_params_t& params, int width, int height);
-        bool prepare_inputs(sd_vid_gen_params_t& params, int width, int height);
-        bool is_valid() const { return valid_; }
-        bool validate_inputs(const sd_img_gen_params_t& params) const;
-        bool validate_inputs(const sd_vid_gen_params_t& params) const;
-        Tensor<float> apply_geometry(const Tensor<float>& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode = ImageResizeMode::Stretch, ops::InterpolateMode default_filter = ops::InterpolateMode::Nearest, ImageTransform* plan_out = nullptr) const;
-        Tensor<float> preprocess_input(sd_image_t image, ImageTarget target, int index = 0, int width = -1, int height = -1);
-    };
-
-}  // namespace sd
-
-#endif  // __SD_RUNTIME_IMAGE_PREPROCESS_H__

src/runtime/preprocessing.hpp

@@ -165,18 +165,16 @@ static inline sd::Tensor<float> convolve_tensor(const sd::Tensor<float>& input,
     return output;
 }
 
-static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& image) {
-    GGML_ASSERT(image.dim() == 4);
-    GGML_ASSERT(image.shape()[2] >= 1);
-    sd::Tensor<float> grayscale({image.shape()[0], image.shape()[1], 1, image.shape()[3]});
-    for (int64_t iy = 0; iy < image.shape()[1]; ++iy) {
-        for (int64_t ix = 0; ix < image.shape()[0]; ++ix) {
-            float gray = preprocessing_get_4d(image, ix, iy, 0, 0);
-            if (image.shape()[2] >= 3) {
-                float g = preprocessing_get_4d(image, ix, iy, 1, 0);
-                float b = preprocessing_get_4d(image, ix, iy, 2, 0);
-                gray    = 0.2989f * gray + 0.5870f * g + 0.1140f * b;
-            }
+static inline sd::Tensor<float> grayscale_tensor(const sd::Tensor<float>& rgb_img) {
+    GGML_ASSERT(rgb_img.dim() == 4);
+    GGML_ASSERT(rgb_img.shape()[2] >= 3);
+    sd::Tensor<float> grayscale({rgb_img.shape()[0], rgb_img.shape()[1], 1, rgb_img.shape()[3]});
+    for (int64_t iy = 0; iy < rgb_img.shape()[1]; ++iy) {
+        for (int64_t ix = 0; ix < rgb_img.shape()[0]; ++ix) {
+            float r    = preprocessing_get_4d(rgb_img, ix, iy, 0, 0);
+            float g    = preprocessing_get_4d(rgb_img, ix, iy, 1, 0);
+            float b    = preprocessing_get_4d(rgb_img, ix, iy, 2, 0);
+            float gray = 0.2989f * r + 0.5870f * g + 0.1140f * b;
             preprocessing_set_4d(grayscale, gray, ix, iy, 0, 0);
         }
     }
@@ -319,12 +317,11 @@ bool preprocess_canny(sd_image_t img, float high_threshold, float low_threshold,
     image_gray              = non_max_supression(G, theta);
     threshold_hystersis(&image_gray, high_threshold, low_threshold, weak, strong);
 
-    const uint32_t color_channels = img.channel == 2 || img.channel == 4 ? img.channel - 1 : img.channel;
     for (uint32_t iy = 0; iy < img.height; ++iy) {
         for (uint32_t ix = 0; ix < img.width; ++ix) {
             float gray = preprocessing_get_4d(image_gray, ix, iy, 0, 0);
             gray       = inverse ? 1.0f - gray : gray;
-            for (uint32_t c = 0; c < color_channels; ++c) {
+            for (uint32_t c = 0; c < img.channel; ++c) {
                 preprocessing_set_4d(image, gray, ix, iy, c, 0);
             }
         }

src/stable-diffusion.cpp

@@ -338,6 +338,8 @@ void sd_ctx_params_init(sd_ctx_params_t* sd_ctx_params) {
     sd_ctx_params->enable_mmap               = false;
     sd_ctx_params->diffusion_flash_attn      = false;
     sd_ctx_params->sage_attn                 = false;
+    sd_ctx_params->sol_attn                  = false;
+    sd_ctx_params->sol_attn_tau              = 1.f;
     sd_ctx_params->linear_scale              = 0.f;
     sd_ctx_params->attn_scale                = 0.f;
     sd_ctx_params->vae_format                = SD_VAE_FORMAT_AUTO;
@@ -394,6 +396,8 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              "flash_attn: %s\n"
              "diffusion_flash_attn: %s\n"
              "sage_attn: %s\n"
+             "sol_attn: %s\n"
+             "sol_attn_tau: %g\n"
              "linear_scale: %g\n"
              "attn_scale: %g\n"
              "vae_format: %s\n",
@@ -434,6 +438,8 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              BOOL_STR(sd_ctx_params->flash_attn),
              BOOL_STR(sd_ctx_params->diffusion_flash_attn),
              BOOL_STR(sd_ctx_params->sage_attn),
+             BOOL_STR(sd_ctx_params->sol_attn),
+             sd_ctx_params->sol_attn_tau,
              sd_ctx_params->linear_scale,
              sd_ctx_params->attn_scale,
              sd_vae_format_name(sd_ctx_params->vae_format));
@@ -630,6 +636,14 @@ struct sd_ctx_t {
     StableDiffusionGGML* sd = nullptr;
 };
 
+static bool sd_version_supports_video_generation(SDVersion version) {
+    return version == VERSION_SVD || sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_lingbot_video(version) || sd_version_is_ltxav(version) || sd_version_is_minimax_h3(version);
+}
+
+static bool sd_version_supports_image_generation(SDVersion version) {
+    return !sd_version_supports_video_generation(version);
+}
+
 sd_ctx_t* new_sd_ctx(const sd_ctx_params_t* sd_ctx_params) {
     sd_ctx_t* sd_ctx = (sd_ctx_t*)malloc(sizeof(sd_ctx_t));
     if (sd_ctx == nullptr) {
@@ -751,20 +765,28 @@ SD_API bool generate_video(sd_ctx_t* sd_ctx,
                            int* num_frames_out,
                            sd_audio_t** audio_out,
                            int* fps_out) {
-    if (frames_out != nullptr)
-        *frames_out = nullptr;
-    if (audio_out != nullptr)
-        *audio_out = nullptr;
-    if (num_frames_out != nullptr)
-        *num_frames_out = 0;
-    if (fps_out != nullptr)
-        *fps_out = 0;
     if (sd_ctx == nullptr || sd_ctx->sd == nullptr || sd_vid_gen_params == nullptr) {
+        if (fps_out != nullptr) {
+            *fps_out = 0;
+        }
         return false;
     }
 
+    if (frames_out != nullptr) {
+        *frames_out = nullptr;
+    }
+    if (audio_out != nullptr) {
+        *audio_out = nullptr;
+    }
+    if (num_frames_out != nullptr) {
+        *num_frames_out = 0;
+    }
+
     StableDiffusionGGML::ExecutionScope execution(*sd_ctx->sd);
     if (!execution.ready) {
+        if (fps_out != nullptr) {
+            *fps_out = 0;
+        }
         return false;
     }
 

src/upscaler.cpp

@@ -111,22 +111,10 @@ bool UpscalerGGML::load_from_file(const std::string& esrgan_path,
 
 sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_tensor) {
     sd::ParallelScope tensor_scope(&tensor_executor);
-    if (input_tensor.empty() || input_tensor.dim() != 4 ||
-        (input_tensor.shape()[2] != 3 && input_tensor.shape()[2] != 4)) {
-        LOG_ERROR("esrgan expects a 4D RGB or RGBA image tensor");
-        return {};
-    }
-
-    const bool has_alpha = input_tensor.shape()[2] == 4;
-    sd::Tensor<float> rgb;
-    if (has_alpha) {
-        rgb = sd::ops::slice(input_tensor, 2, 0, 3);
-    }
-    const sd::Tensor<float>& model_input = has_alpha ? rgb : input_tensor;
     sd::Tensor<float> upscaled;
     const int scale = esrgan_upscaler->config.scale;
     if (tile_size <= 0 || (input_tensor.shape()[0] <= tile_size && input_tensor.shape()[1] <= tile_size)) {
-        upscaled = esrgan_upscaler->compute(n_threads, model_input);
+        upscaled = esrgan_upscaler->compute(n_threads, input_tensor);
     } else {
         auto on_processing = [&](const sd::Tensor<float>& input_tile) -> sd::Tensor<float> {
             auto output_tile = esrgan_upscaler->compute(n_threads, input_tile);
@@ -137,7 +125,7 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
             return output_tile;
         };
 
-        upscaled = process_tiles_2d(model_input,
+        upscaled = process_tiles_2d(input_tensor,
                                     static_cast<int>(input_tensor.shape()[0] * scale),
                                     static_cast<int>(input_tensor.shape()[1] * scale),
                                     scale,
@@ -153,14 +141,6 @@ sd::Tensor<float> UpscalerGGML::upscale_tensor(const sd::Tensor<float>& input_te
         LOG_ERROR("esrgan compute failed");
         return {};
     }
-    if (has_alpha) {
-        auto alpha       = sd::ops::slice(input_tensor, 2, 3, 4);
-        auto alpha_shape = alpha.shape();
-        alpha_shape[0]   = upscaled.shape()[0];
-        alpha_shape[1]   = upscaled.shape()[1];
-        alpha            = sd::ops::interpolate(alpha, alpha_shape, sd::ops::InterpolateMode::Bilinear);
-        upscaled         = sd::ops::concat(upscaled, alpha, 2);
-    }
     return upscaled;
 }