feat: add native CUDA Sol-Attn support

docs/backend.md

@@ -156,11 +156,8 @@ the runner's graph-cut capacity checks.
 
 Runtime capacity checks also leave 512 MiB of currently free device memory for
 backend scratch buffers and pipelines, including with explicit backend assignments.
-They cap free-memory reports by the device's total memory minus tracked
-resident allocations. Vulkan reports exceeding total memory are rejected because
-its heap-budget subtraction can underflow. Other backends use the cap instead of
-treating such reports as zero free memory. Failed checks log the reported free and
-total memory alongside tracked weight and runtime allocations.
+They cap stale free-memory reports by the device's total memory minus tracked
+resident allocations and reject reports that exceed the device's total memory.
 
 Components are considered in `diffusion`, `te`, `vae` order so that repeatedly
 used diffusion weights have priority. Each component's weights use the first

docs/qwen_image_2.1.md

@@ -39,21 +39,3 @@ Pass the reference image with `-r` and describe the edit in `-p`. Vision weights
 ```
 
 For multiple reference images, repeat `-r` in the desired order, for example `-r first.png -r second.png`.
-
-### Alpha channel
-
-This model supports alpha channel output. As the model determines whether to output a regular image or with transparency through the prompt, according to [official recommendation](https://github.com/QwenLM/Qwen-Image-2.1#transparent-image-generation-rgba), use the following prompt format for better results:
-
-> `This is an RGBA image with transparency. <your description>. The image has alpha channel and the background is transparent.`
-
-Since transparency is decided by the prompt rather than by the input or an explicit switch, the same format applies equally to editing, whether or not the reference image itself has an alpha channel. Note that alpha is kept only in `.png` and `.webp` outputs; saving as `.jpg` drops the transparency.
-
-Here are some examples ran with Q6_K quantization:
-| Input | Prompt | Output |
-| --- | --- | --- |
-| ![Qwen Image 2.1 alpha input example 1](../assets/qwen/qwen-image-2.1-alpha-in1.png) | This is an RGBA image with transparency. Replace the text "BLOOM" with "Qwen Image 2.1", keeping the same font of the original text. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 1](../assets/qwen/qwen-image-2.1-alpha-out1.png) |
-| ![Qwen Image 2.1 alpha input example 2](../assets/logo.png) | This is an RGBA image with transparency. Remove the background of the image, keeping only the text and cat. The image has alpha channel and the background is transparent. | ![Qwen Image 2.1 alpha output example 2](../assets/qwen/qwen-image-2.1-alpha-out2.png) |
-
-### Other features
-
-Other features of the model could be found on the [model card from QwenLM/Qwen-Image-2.1 repo](https://github.com/QwenLM/Qwen-Image-2.1), including 2 finetuned prompt rewriting Qwen3.5-9B model.

docs/sol_attention.md

@@ -0,0 +1,106 @@
+# Sol-Attn
+
+`--sol-attn` enables native CUDA Sol-Attn in the diffusion model, including the
+high-noise diffusion model when present. It uses the shared attention dispatcher
+without classifying tokens as text, images, or video. Python, PyTorch, Triton,
+and CuTe DSL are not needed to build or run it.
+
+This implementation follows the diagonal-threshold algorithm in
+[NVlabs/Sana's Sol-Attn](https://github.com/NVlabs/Sana/tree/sol-engine/techniques/sparse_backends/sol_attn).
+It summarizes 64-token KV blocks, selects exact blocks using proxy scores and
+an online threshold, and approximates the remaining blocks using their K means
+and V sums. Adjacent blocks remain exact. Both contributions share an online
+softmax normalizer. Q/K/V and probability tiles use BF16 Tensor Cores with FP32
+accumulation; the BF16 result is returned through the existing FP32 interface.
+
+## Build
+
+Use patched GGML, CUDA Toolkit 12.0 or newer, and an NVIDIA GPU with compute
+capability 8.0 or newer. Compile kernels for the target GPU:
+
+```sh
+cmake -S . -B build -DSD_CUDA=ON -DSD_USE_UPSTREAM_GGML=OFF
+cmake --build build --config Release
+```
+
+The feature is compiled with the CUDA backend; no separate build option is
+required. Upstream GGML and non-CUDA backends do not support it. A system GGML
+must provide the matching patched API and CUDA implementation. Tensor-parallel
+row splitting is not supported; layer splitting requires supported devices.
+
+## Use
+
+Add `--sol-attn` to an existing generation command:
+
+```sh
+sd-cli ... --sol-attn
+sd-cli ... --sol-attn --sol-attn-tau 1.0
+```
+
+The default threshold coefficient is `1.0`. Larger coefficients select fewer
+blocks for exact attention. The coefficient must be finite; zero does not mean
+dense attention. Omit `--sol-attn` to disable the feature.
+
+The native kernel supports unmasked, noncausal attention with head dimension
+128, equal Q/K/V sequence lengths and head counts, and multiple batches. Other
+attention operations fall back to FlashAttention when available, then ordinary
+attention. Existing attention scaling overrides remain effective. `--fa` and
+`--diffusion-fa` may be used together with Sol-Attn; `--sage-attn` is mutually
+exclusive. Text encoders and VAEs retain their existing attention selection.
+
+Initialization reports an error if the requested diffusion backend cannot run
+Sol-Attn. Graph logs report the number of Sol-Attn and FlashAttention nodes and
+warn when no Sol-Attn nodes are selected. CUDA execution errors are not silently
+converted into dense attention.
+
+This is approximate attention. Validate quality and end-to-end speed with the
+same prompt, seed, dimensions, frame count, and sampling settings. Include
+packing, preprocessing, offload, and decode time in comparisons. Short sequences
+may not benefit. Upstream combined pipeline speedups are not measurements of
+this native kernel. Exact-covariance thresholds, text sinks, Morton ordering,
+and step/layer schedules are not implemented.
+
+## Validation
+
+On an RTX 4090 with CUDA 12.4, Wan 2.1 T2V 1.3B was tested at 832x480,
+33 frames, 20 Euler steps, seed 42, CFG 6, and flow shift 3, using the prompt
+`a lovely cat` and the same negative prompt for every run:
+
+| Attention | Sampling time | Total process time |
+| --- | ---: | ---: |
+| FlashAttention | 45.73 s | 74.63 s |
+| Sol-Attn, tau 1 | 37.17 s | 66.20 s |
+| Sol-Attn, tau 0 | 40.66 s | 68.50 s |
+
+These are single-run measurements. The graph selected 30 Sol-Attn nodes and
+30 FlashAttention nodes. At tau 1, sampled video frames showed washed-out
+colors and reduced detail. Tau 0 improved clarity in this example, but still
+changed the composition. Neither setting guarantees the baseline's quality.
+For this Wan command, `--sol-attn --sol-attn-tau 0` is a more conservative
+starting point. In the one-frame case, tau 1 increased warm sampling time from
+0.140 to 0.148 seconds per step.
+
+Validation also covered 15 numerical reference cases, 11 layout/scaling/fallback
+cases, CUDA memory checking, and 36 existing SageAttention regression cases.
+CLI and server CUDA builds and the upstream GGML CPU library build passed.
+Other GPU architectures, multi-GPU execution, and other models have not been
+tested.
+
+## Library API
+
+Configure Sol-Attn in `sd_ctx_params_t` before creating the context:
+
+```cpp
+sd_ctx_params_t params;
+sd_ctx_params_init(&params);
+// Set model paths and other context options here.
+params.sol_attn = true;
+params.sol_attn_tau = 1.0f;
+sd_ctx_t* ctx = new_sd_ctx(&params);
+```
+
+`sd_ctx_params_init` defaults `sol_attn` to false and `sol_attn_tau` to 1.0.
+`new_sd_ctx` returns null for a nonfinite threshold, unavailable requested
+backends, or a conflict with SageAttention. The context owns a copy of these
+settings; changing the input structure after creation does not reconfigure it.
+Applications must be rebuilt against the updated `sd_ctx_params_t` definition.

docs/troubleshooting.md

@@ -1,14 +1,5 @@
 # Troubleshooting
 
-## Video model used in image generation mode
-
-If generation reports that a model cannot be run with `generate_image()`, add
-`--mode vid_gen` to the CLI command. `--video-frames` alone does not select video
-mode. Video models require this mode even when generating a single frame.
-Library callers must use `generate_video()` for these models; use
-`sd_ctx_supports_image_generation()` and `sd_ctx_supports_video_generation()` to
-check the available generation modes.
-
 ## Completely black or white images or videos / NaNs
 
 Some ggml backends can encounter numerical overflow during inference, producing

docs/wan.md

@@ -1,7 +1,5 @@
 # How to Use
 
-Wan models require `-M vid_gen`, including single-frame generation. `--video-frames` alone does not select video mode. Library callers must use `generate_video()` instead of `generate_image()`.
-
 ## Download weights
 
 - Download Wan

examples/cli/README.md

@@ -25,3 +25,7 @@ Metadata mode inspects PNG/JPEG container metadata without loading any model:
 
 For completely black or white images or videos, NaNs, and the `--linear-scale` /
 `--attn-scale` workaround, see [Troubleshooting](../../docs/troubleshooting.md).
+
+For native CUDA sparse attention in the diffusion model, use `--sol-attn`.
+See [Sol-Attn](../../docs/sol_attention.md) for requirements, supported shapes,
+and the `--sol-attn-tau` threshold coefficient.

examples/cli/main.cpp

@@ -357,8 +357,7 @@ bool load_images_from_dir(const std::string dir,
             LOG_VERBOSE("load image %zu from '%s'", images.size(), path.c_str());
             int width             = 0;
             int height            = 0;
-            int loaded_channel    = 0;
-            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, expected_width, expected_height);
+            uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, expected_width, expected_height);
             if (image_buffer == nullptr) {
                 LOG_ERROR("load image from '%s' failed", path.c_str());
                 return false;
@@ -366,7 +365,7 @@ bool load_images_from_dir(const std::string dir,
 
             images.emplace_back(sd_image_t{(uint32_t)width,
                                            (uint32_t)height,
-                                           (uint32_t)loaded_channel,
+                                           3,
                                            image_buffer});
 
             if (max_image_num > 0 && static_cast<int>(images.size()) >= max_image_num) {
@@ -782,8 +781,7 @@ int main(int argc, const char* argv[]) {
     };
 
     if (gen_params.init_image_path.size() > 0) {
-        const bool native_init = cli_params.mode == IMG_GEN || cli_params.mode == ADETAILER;
-        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, true, native_init ? 0 : 3)) {
+        if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image)) {
             return 1;
         }
     }
@@ -797,8 +795,8 @@ int main(int argc, const char* argv[]) {
     if (gen_params.ref_image_paths.size() > 0) {
         gen_params.ref_images.clear();
         for (auto& path : gen_params.ref_image_paths) {
-            SDImageOwner ref_image({0, 0, 0, nullptr});
-            if (!load_image_and_update_size(path, ref_image, false, 0)) {
+            SDImageOwner ref_image({0, 0, 3, nullptr});
+            if (!load_image_and_update_size(path, ref_image, false)) {
                 return 1;
             }
             gen_params.ref_images.push_back(std::move(ref_image));

examples/common/common.cpp

@@ -622,6 +622,10 @@ ArgOptions SDContextParams::get_options() {
          "--sage-attn",
          "use native CUDA SageAttention in the diffusion model, with flash/default attention fallback",
          true, &sage_attn},
+        {"",
+         "--sol-attn",
+         "use native CUDA Sol-Attn in the diffusion model, with flash/default attention fallback",
+         true, &sol_attn},
         {"",
          "--diffusion-conv-direct",
          "use ggml_conv2d_direct in the diffusion model",
@@ -719,6 +723,8 @@ ArgOptions SDContextParams::get_options() {
         return 1;
     };
 
+    options.float_options.push_back({"", "--sol-attn-tau", "Sol-Attn routing threshold coefficient (default: 1; higher selects fewer exact blocks)", &sol_attn_tau});
+
     options.manual_options = {
         {"",
          "--linear-scale",
@@ -822,6 +828,14 @@ bool SDContextParams::resolve(SDMode mode) {
 }
 
 bool SDContextParams::validate(SDMode mode) {
+    if (sol_attn && sage_attn) {
+        LOG_ERROR("--sol-attn and --sage-attn cannot be enabled together");
+        return false;
+    }
+    if (!std::isfinite(sol_attn_tau)) {
+        LOG_ERROR("--sol-attn-tau must be finite");
+        return false;
+    }
     if (mode == CONVERT) {
         const bool has_convert_input = model_path.length() != 0 ||
                                        clip_l_path.length() != 0 ||
@@ -943,6 +957,8 @@ std::string SDContextParams::to_string() const {
         << "  flash_attn: " << (flash_attn ? "true" : "false") << ",\n"
         << "  diffusion_flash_attn: " << (diffusion_flash_attn ? "true" : "false") << ",\n"
         << "  sage_attn: " << (sage_attn ? "true" : "false") << ",\n"
+        << "  sol_attn: " << (sol_attn ? "true" : "false") << ",\n"
+        << "  sol_attn_tau: " << sol_attn_tau << ",\n"
         << "  linear_scale: " << linear_scale << ",\n"
         << "  attn_scale: " << attn_scale << ",\n"
         << "  diffusion_conv_direct: " << (diffusion_conv_direct ? "true" : "false") << ",\n"
@@ -1001,6 +1017,8 @@ sd_ctx_params_t SDContextParams::to_sd_ctx_params_t(bool taesd_preview) {
     sd_ctx_params.flash_attn                      = flash_attn;
     sd_ctx_params.diffusion_flash_attn            = diffusion_flash_attn;
     sd_ctx_params.sage_attn                       = sage_attn;
+    sd_ctx_params.sol_attn                        = sol_attn;
+    sd_ctx_params.sol_attn_tau                    = sol_attn_tau;
     sd_ctx_params.linear_scale                    = linear_scale;
     sd_ctx_params.attn_scale                      = attn_scale;
     sd_ctx_params.tae_preview_only                = taesd_preview;
@@ -1754,7 +1772,7 @@ ArgOptions SDGenerationParams::get_options() {
          on_scm_policy_arg},
         {"",
          "--vae-tile-size",
-         "tile size for vae tiling in latent units, not image pixels, format [X]x[Y] (default: 32x32)",
+         "tile size for vae tiling, format [X]x[Y] (default: 32x32)",
          on_tile_size_arg},
         {"",
          "--vae-relative-tile-size",
@@ -1848,22 +1866,20 @@ bool decode_base64_image(const std::string& encoded_input,
         return false;
     }
 
-    int decoded_width    = 0;
-    int decoded_height   = 0;
-    int resolved_channel = target_channels;
-    uint8_t* raw_data    = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
-                                                  static_cast<int>(image_bytes.size()),
-                                                  decoded_width,
-                                                  decoded_height,
-                                                  resolved_channel,
-                                                  expected_width,
-                                                  expected_height,
-                                                  target_channels);
+    int decoded_width  = 0;
+    int decoded_height = 0;
+    uint8_t* raw_data  = load_image_from_memory(reinterpret_cast<const char*>(image_bytes.data()),
+                                                static_cast<int>(image_bytes.size()),
+                                                decoded_width,
+                                                decoded_height,
+                                                expected_width,
+                                                expected_height,
+                                                target_channels);
     if (raw_data == nullptr) {
         return false;
     }
 
-    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)resolved_channel, raw_data});
+    out_image.reset({(uint32_t)decoded_width, (uint32_t)decoded_height, (uint32_t)target_channels, raw_data});
     return true;
 }
 
@@ -2217,7 +2233,7 @@ bool SDGenerationParams::from_json_str(
         LOG_ERROR("invalid lora");
         return false;
     }
-    if (!parse_image_json_field(j, "init_image", 0, width, height, init_image)) {
+    if (!parse_image_json_field(j, "init_image", 3, width, height, init_image)) {
         LOG_ERROR("invalid init_image");
         return false;
     }
@@ -2225,12 +2241,7 @@ bool SDGenerationParams::from_json_str(
         LOG_ERROR("invalid end_image");
         return false;
     }
-    if (!parse_image_array_json_field(j,
-                                      "ref_images",
-                                      0,
-                                      auto_resize_ref_image ? width : 0,
-                                      auto_resize_ref_image ? height : 0,
-                                      ref_images)) {
+    if (!parse_image_array_json_field(j, "ref_images", 3, width, height, ref_images)) {
         LOG_ERROR("invalid ref_images");
         return false;
     }

examples/common/common.h

@@ -171,6 +171,8 @@ struct SDContextParams {
     bool flash_attn            = false;
     bool diffusion_flash_attn  = false;
     bool sage_attn             = false;
+    bool sol_attn              = false;
+    float sol_attn_tau         = 1.f;
     bool diffusion_conv_direct = false;
     bool vae_conv_direct       = false;
 

examples/common/media_io.cpp

@@ -261,10 +261,6 @@ uint8_t* decode_webp_image_to_buffer(const uint8_t* data,
     height               = features.height;
     source_channel_count = features.has_alpha ? 4 : 3;
 
-    if (expected_channel == 0) {
-        expected_channel = source_channel_count;
-    }
-
     const size_t pixel_count = static_cast<size_t>(width) * static_cast<size_t>(height);
 
     if (expected_channel == 1) {
@@ -485,8 +481,7 @@ uint8_t* load_image_common(bool from_memory,
                            int& height,
                            int expected_width,
                            int expected_height,
-                           int expected_channel,
-                           int& out_channel) {
+                           int expected_channel) {
     const char* image_path;
     FreeUniquePtr<uint8_t> image_buffer;
     int source_channel_count = 0;
@@ -543,32 +538,6 @@ uint8_t* load_image_common(bool from_memory,
         LOG_ERROR("load image from '%s' failed", image_path);
         return nullptr;
     }
-    if (expected_channel == 0) {
-        expected_channel = source_channel_count == 2 ? 4 : (source_channel_count == 1 ? 3 : source_channel_count);
-        if (expected_channel != source_channel_count) {
-            FreeUniquePtr<uint8_t> promoted((uint8_t*)malloc((size_t)width * height * expected_channel));
-            if (promoted == nullptr) {
-                LOG_ERROR("error: allocate memory for channel promotion, image_path = %s", image_path);
-                return nullptr;
-            }
-            const size_t pixel_count = (size_t)width * (size_t)height;
-            for (size_t i = 0; i < pixel_count; ++i) {
-                if (source_channel_count == 1) {
-                    promoted.get()[i * 3 + 0] = image_buffer.get()[i];
-                    promoted.get()[i * 3 + 1] = image_buffer.get()[i];
-                    promoted.get()[i * 3 + 2] = image_buffer.get()[i];
-                } else {
-                    promoted.get()[i * 4 + 0] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 1] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 2] = image_buffer.get()[i * 2];
-                    promoted.get()[i * 4 + 3] = image_buffer.get()[i * 2 + 1];
-                }
-            }
-            image_buffer         = std::move(promoted);
-            source_channel_count = expected_channel;
-        }
-    }
-    // stb reports the source channel count even when it converts the output.
     if (source_channel_count < expected_channel) {
         fprintf(stderr,
                 "the number of channels for the input image must be >= %d,"
@@ -628,7 +597,7 @@ uint8_t* load_image_common(bool from_memory,
         }
         stbir_resize(image_buffer.get(), width, height, 0,
                      resized_image_buffer.get(), expected_width, expected_height, 0, STBIR_TYPE_UINT8,
-                     expected_channel, expected_channel == 4 ? 3 : STBIR_ALPHA_CHANNEL_NONE, 0,
+                     expected_channel, STBIR_ALPHA_CHANNEL_NONE, 0,
                      STBIR_EDGE_CLAMP, STBIR_EDGE_CLAMP,
                      STBIR_FILTER_BOX, STBIR_FILTER_BOX,
                      STBIR_COLORSPACE_SRGB, nullptr);
@@ -636,7 +605,6 @@ uint8_t* load_image_common(bool from_memory,
         height       = expected_height;
         image_buffer = std::move(resized_image_buffer);
     }
-    out_channel = expected_channel;
     return image_buffer.release();
 }
 
@@ -809,11 +777,10 @@ bool write_image_to_file(const std::string& path,
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
-                              int& out_channel,
                               int expected_width,
                               int expected_height,
                               int expected_channel) {
-    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, out_channel);
+    return load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
 }
 
 bool load_sd_image_from_file(sd_image_t* image,
@@ -823,14 +790,13 @@ bool load_sd_image_from_file(sd_image_t* image,
                              int expected_channel) {
     int width;
     int height;
-    int resolved_channel = expected_channel;
-    image->data          = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel, resolved_channel);
+    image->data = load_image_common(false, image_path, 0, width, height, expected_width, expected_height, expected_channel);
     if (image->data == nullptr) {
         return false;
     }
     image->width   = width;
     image->height  = height;
-    image->channel = resolved_channel;
+    image->channel = expected_channel;
     return true;
 }
 
@@ -838,11 +804,10 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
-                                int& out_channel,
                                 int expected_width,
                                 int expected_height,
                                 int expected_channel) {
-    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel, out_channel);
+    return load_image_common(true, image_bytes, len, width, height, expected_width, expected_height, expected_channel);
 }
 
 static void append_avi_metadata(std::vector<uint8_t>& data, const std::string& parameters) {

examples/common/media_io.h

@@ -32,12 +32,9 @@ bool write_image_to_file(const std::string& path,
                          const std::string& parameters = "",
                          int quality                   = 90);
 
-// expected_channel == 0 preserves native channels (grayscale -> RGB, gray+alpha -> RGBA).
-// out_channel receives the output channel count.
 uint8_t* load_image_from_file(const char* image_path,
                               int& width,
                               int& height,
-                              int& out_channel,
                               int expected_width   = 0,
                               int expected_height  = 0,
                               int expected_channel = 3);
@@ -52,7 +49,6 @@ uint8_t* load_image_from_memory(const char* image_bytes,
                                 int len,
                                 int& width,
                                 int& height,
-                                int& out_channel,
                                 int expected_width   = 0,
                                 int expected_height  = 0,
                                 int expected_channel = 3);

examples/server/api.md

@@ -735,15 +735,12 @@ Any image field accepts:
 
 Channel expectations:
 
-- `init_image`: native channels (3 or 4); alpha is preserved and applied per model
-- `ref_images[]`: native channels (3 or 4); alpha is preserved and applied per model
+- `init_image`: 3 channels
+- `ref_images[]`: 3 channels
 - `control_image`: 3 channels
 - `ip_adapter_image`: 3 channels
 - `mask_image`: 1 channel
 
-Models that support RGBA (e.g. Qwen-Image 2.1) use the alpha channel of `init_image`
-and `ref_images[]`. RGB-only models drop it, so sending RGBA is safe for every model.
-
 If omitted or null:
 
 - single-image fields map to an empty `sd_image_t`

examples/server/routes_openai.cpp

@@ -158,46 +158,26 @@ static bool build_openai_edit_request(const httplib::Request& req,
     request.gen_params.batch_count = n;
 
     for (auto& bytes : images_bytes) {
-        int img_w            = 0;
-        int img_h            = 0;
-        int resolved_channel = 0;
-        uint8_t* raw_pixels  = load_image_from_memory(
-             reinterpret_cast<const char*>(bytes.data()),
-             static_cast<int>(bytes.size()),
-             img_w, img_h, resolved_channel,
-             0, 0, 0);
+        int img_w           = 0;
+        int img_h           = 0;
+        uint8_t* raw_pixels = load_image_from_memory(
+            reinterpret_cast<const char*>(bytes.data()),
+            static_cast<int>(bytes.size()),
+            img_w, img_h,
+            width, height, 3);
         if (raw_pixels == nullptr) {
             continue;
         }
 
-        const bool is_first_ref_image = request.gen_params.ref_images.empty();
-        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, (uint32_t)resolved_channel, raw_pixels});
+        SDImageOwner image_owner({(uint32_t)img_w, (uint32_t)img_h, 3, raw_pixels});
         request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
-
-        if (is_first_ref_image) {
-            int init_w = 0;
-            int init_h = 0;
-            if (request.gen_params.width_and_height_are_set()) {
-                init_w = request.gen_params.width;
-                init_h = request.gen_params.height;
-            }
-
-            int init_img_w            = 0;
-            int init_img_h            = 0;
-            int init_resolved_channel = 0;
-            uint8_t* init_pixels     = load_image_from_memory(
-                 reinterpret_cast<const char*>(bytes.data()),
-                 static_cast<int>(bytes.size()),
-                 init_img_w, init_img_h, init_resolved_channel,
-                 init_w, init_h, 0);
-            if (init_pixels != nullptr) {
-                request.gen_params.init_image.reset({(uint32_t)init_img_w, (uint32_t)init_img_h, (uint32_t)init_resolved_channel, init_pixels});
-            }
-        }
-
         request.gen_params.ref_images.push_back(std::move(image_owner));
     }
 
+    if (!request.gen_params.ref_images.empty()) {
+        request.gen_params.init_image = request.gen_params.ref_images.front();
+    }
+
     if (!mask_bytes.empty()) {
         int expected_width  = 0;
         int expected_height = 0;
@@ -205,14 +185,13 @@ static bool build_openai_edit_request(const httplib::Request& req,
             expected_width  = request.gen_params.width;
             expected_height = request.gen_params.height;
         }
-        int mask_w       = 0;
-        int mask_h       = 0;
-        int mask_channel = 0;
+        int mask_w = 0;
+        int mask_h = 0;
 
         uint8_t* mask_raw = load_image_from_memory(
             reinterpret_cast<const char*>(mask_bytes.data()),
             static_cast<int>(mask_bytes.size()),
-            mask_w, mask_h, mask_channel,
+            mask_w, mask_h,
             expected_width, expected_height, 1);
         request.gen_params.mask_image.reset({(uint32_t)mask_w, (uint32_t)mask_h, 1, mask_raw});
         const sd_image_t& mask_image = request.gen_params.mask_image.get();

examples/server/routes_sdapi.cpp

@@ -199,7 +199,7 @@ static bool build_sdapi_img_gen_request(const json& j,
 
         if (j.contains("init_images") && j["init_images"].is_array() && !j["init_images"].empty()) {
             if (decode_base64_image(j["init_images"][0].get<std::string>(),
-                                    0,
+                                    3,
                                     expected_width,
                                     expected_height,
                                     request.gen_params.init_image)) {
@@ -243,13 +243,9 @@ static bool build_sdapi_img_gen_request(const json& j,
             }
             SDImageOwner image_owner;
             if (decode_base64_image(extra_image.get<std::string>(),
-                                    0,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.width
-                                        : 0,
-                                    request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
-                                        ? request.gen_params.height
-                                        : 0,
+                                    3,
+                                    request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0,
+                                    request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0,
                                     image_owner)) {
                 const sd_image_t& image = image_owner.get();
                 request.gen_params.set_width_and_height_if_unset(image.width, image.height);

include/stable-diffusion.h

@@ -247,6 +247,8 @@ typedef struct {
     float attn_scale;                // Override flash-attention K/V scaling; 0 keeps the model default
     const char* tokenizer;           // tokenizer.json path or main=FILE,clip-l=FILE,clip-g=FILE assignments; required for PiD and Lens
     bool sage_attn;
+    bool sol_attn;
+    float sol_attn_tau;
 } sd_ctx_params_t;
 
 typedef struct {

src/conditioning/conditioner.hpp

@@ -2219,10 +2219,7 @@ struct LLMEmbedder : public Conditioner {
                                           false,
                                           deepstack_image_embeds,
                                           image_grids);
-        if (hidden_states.empty()) {
-            LOG_ERROR("LLM prompt encoding failed");
-            return {};
-        }
+        GGML_ASSERT(!hidden_states.empty());
         hidden_states = apply_token_weights(std::move(hidden_states), weights);
         GGML_ASSERT(hidden_states.shape()[1] > prompt_template_encode_start_idx);
 

src/core/backend_fit.cpp

@@ -478,11 +478,7 @@ namespace sd::backend_fit {
         return true;
     }
 
-    bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling, ggml_status status) {
-        // Execution failures can leave the device unusable; tiling only helps with allocation failures.
-        if (status != GGML_STATUS_ALLOC_FAILED) {
-            return false;
-        }
+    bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling) {
         const char* retry_mode = nullptr;
         if (prefer_temporal_tiling && !tiling_params.temporal_tiling) {
             tiling_params.temporal_tiling = true;
@@ -502,7 +498,7 @@ namespace sd::backend_fit {
             return false;
         }
 
-        LOG_WARN("VAE decode ran out of memory; retrying with %s tiling",
+        LOG_WARN("VAE decode failed (likely out of memory); retrying with %s tiling",
                  retry_mode);
         return true;
     }

src/core/backend_fit.h

@@ -16,8 +16,7 @@ namespace sd::backend_fit {
                               std::string& params_spec);
 
     bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
-                                         bool prefer_temporal_tiling,
-                                         ggml_status status);
+                                         bool prefer_temporal_tiling);
 
 }  // namespace sd::backend_fit
 

src/core/ggml_extend.cpp

@@ -623,7 +623,9 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
                                     bool skip_reshape,
                                     bool flash_attn,
                                     float kv_scale,
-                                    bool sage_attn) {  // avoid overflow
+                                    bool sage_attn,
+                                    bool sol_attn,
+                                    float sol_attn_tau) {  // avoid overflow
     int64_t L_q;
     int64_t L_k;
     int64_t C;
@@ -715,7 +717,23 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
     };
 
 #ifndef SD_USE_UPSTREAM_GGML
-    if (sage_attn && mask == nullptr && d_head > 0 && d_head <= 128) {
+    if (sol_attn && mask == nullptr && d_head == 128 && L_q == L_k && n_head == n_kv_head) {
+        auto q_in = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, q->type == GGML_TYPE_F32 ? q : ggml_cast(ctx, q, GGML_TYPE_F32)), d_head, L_q, n_head, N);
+        auto k_in = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, k->type == GGML_TYPE_F32 ? k : ggml_cast(ctx, k, GGML_TYPE_F32)), d_head, L_k, n_kv_head, N);
+        auto v_in = ggml_ext_cont(ctx, ggml_permute(ctx, v, 0, 2, 1, 3));
+        if (v_in->type != GGML_TYPE_F32) {
+            v_in = ggml_cast(ctx, v_in, GGML_TYPE_F32);
+        }
+        if (kv_scale != 1.0f) {
+            k_in = ggml_ext_scale(ctx, k_in, kv_scale);
+            v_in = ggml_ext_scale(ctx, v_in, kv_scale);
+        }
+        auto out = ggml_sol_attn(ctx, q_in, k_in, v_in, scale / kv_scale, sol_attn_tau);
+        if (ggml_backend_supports_op(backend, out)) {
+            kqv = kv_scale != 1.0f ? ggml_ext_scale(ctx, out, 1.0f / kv_scale) : out;
+        }
+    }
+    if (kqv == nullptr && sage_attn && mask == nullptr && d_head > 0 && d_head <= 128) {
         auto q_in                 = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, q->type == GGML_TYPE_F32 ? q : ggml_cast(ctx, q, GGML_TYPE_F32)), d_head, L_q, n_head, N);
         auto k_in                 = ggml_reshape_4d(ctx, ggml_ext_cont(ctx, k->type == GGML_TYPE_F32 ? k : ggml_cast(ctx, k, GGML_TYPE_F32)), d_head, L_k, n_kv_head, N);
         auto v_in                 = ggml_ext_cont(ctx, ggml_permute(ctx, v, 0, 2, 1, 3));
@@ -744,7 +762,7 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
     }
 #endif
 
-    if (kqv == nullptr && (flash_attn || sage_attn)) {
+    if (kqv == nullptr && (flash_attn || sage_attn || sol_attn)) {
         // LOG_VERBOSE("attention_ext L_q:%d L_k:%d n_head:%d C:%d d_head:%d N:%d", L_q, L_k, n_head, C, d_head, N);
         bool can_use_flash_attn = true;
         if (mask != nullptr) {

src/core/ggml_extend.h

@@ -217,11 +217,13 @@ ggml_tensor* ggml_ext_attention_ext(ggml_context* ctx,
                                     ggml_tensor* k,
                                     ggml_tensor* v,
                                     int64_t n_head,
-                                    ggml_tensor* mask = nullptr,
-                                    bool skip_reshape = false,
-                                    bool flash_attn   = false,
-                                    float kv_scale    = 1.0f,
-                                    bool sage_attn    = false);
+                                    ggml_tensor* mask  = nullptr,
+                                    bool skip_reshape  = false,
+                                    bool flash_attn    = false,
+                                    float kv_scale     = 1.0f,
+                                    bool sage_attn     = false,
+                                    bool sol_attn      = false,
+                                    float sol_attn_tau = 1.0f);
 
 ggml_tensor* ggml_ext_layer_norm(ggml_context* ctx,
                                  ggml_tensor* x,

src/core/ggml_runner.cpp

@@ -25,7 +25,7 @@ ggml_tensor* ggml_ext_attention_ext(GGMLRunnerContext* ctx,
     if (ctx->attn_scale > 0.f) {
         kv_scale = ctx->attn_scale;
     }
-    return ggml_ext_attention_ext(ctx->ggml_ctx, ctx->backend, q, k, v, n_head, mask, skip_reshape, flash_attn, kv_scale, ctx->sage_attn_enabled);
+    return ggml_ext_attention_ext(ctx->ggml_ctx, ctx->backend, q, k, v, n_head, mask, skip_reshape, flash_attn, kv_scale, ctx->sage_attn_enabled, ctx->sol_attn_enabled, ctx->sol_attn_tau);
 }
 
 void GGMLRunner::alloc_params_ctx() {
@@ -164,6 +164,22 @@ ggml_cgraph* GGMLRunner::get_compute_graph(get_graph_cb_t get_graph) {
         }
     }
     prepare_build_in_tensor_after(gf);
+#ifndef SD_USE_UPSTREAM_GGML
+    if (sol_attn_enabled && !sol_attn_graph_logged) {
+        int sol_nodes   = 0;
+        int flash_nodes = 0;
+        for (int i = 0; i < ggml_graph_n_nodes(gf); ++i) {
+            const auto op = ggml_graph_node(gf, i)->op;
+            sol_nodes += op == GGML_OP_SOL_ATTN;
+            flash_nodes += op == GGML_OP_FLASH_ATTN_EXT;
+        }
+        LOG_INFO("Sol-Attn graph: %d Sol-Attn nodes, %d FlashAttention nodes", sol_nodes, flash_nodes);
+        if (sol_nodes == 0) {
+            LOG_WARN("This graph has no attention operations supported by Sol-Attn");
+        }
+        sol_attn_graph_logged = true;
+    }
+#endif
     return gf;
 }
 
@@ -521,6 +537,8 @@ GGMLRunnerContext GGMLRunner::get_context() {
     runner_ctx.backend               = runtime_backend;
     runner_ctx.flash_attn_enabled    = flash_attn_enabled;
     runner_ctx.sage_attn_enabled     = sage_attn_enabled;
+    runner_ctx.sol_attn_enabled      = sol_attn_enabled;
+    runner_ctx.sol_attn_tau          = sol_attn_tau;
     runner_ctx.linear_scale          = linear_scale;
     runner_ctx.attn_scale            = attn_scale;
     runner_ctx.conv2d_direct_enabled = conv2d_direct_enabled;
@@ -590,7 +608,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
                                                      bool auto_runner_end,
                                                      bool no_return,
                                                      const std::function<bool()>& read_outputs) {
-    last_compute_status_ = GGML_STATUS_FAILED;
     if (graph_active_) {
         LOG_ERROR("%s does not support reentrant graph execution", get_desc().c_str());
         return std::nullopt;
@@ -614,9 +631,7 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
         GGMLRunner& runner;
         const bool& success;
         ~GraphEndGuard() {
-            if (!runner.workspace_.segment_end()) {
-                runner.last_compute_status_ = GGML_STATUS_FAILED;
-            }
+            runner.workspace_.segment_end();
             runner.cache_.graph_end(false);
             runner.cut_cache_.clear();
             runner.free_compute_ctx();
@@ -645,7 +660,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
     try {
         output = execute_graph(graph, n_threads, no_return, read_outputs);
     } catch (const std::exception& error) {
-        last_compute_status_ = GGML_STATUS_FAILED;
         LOG_ERROR("%s graph execution failed on %s: %s", get_desc().c_str(),
                   ggml_backend_name(runtime_backend), error.what());
         return std::nullopt;
@@ -653,7 +667,6 @@ std::optional<sd::Tensor<float>> GGMLRunner::compute(get_graph_cb_t get_graph,
     success = output.has_value();
     if (success) {
         cache_.graph_end(true);
-        last_compute_status_ = GGML_STATUS_SUCCESS;
     }
     return output;
 }
@@ -771,7 +784,6 @@ bool GGMLRunner::execute_segment(ggml_cgraph* graph, int n_threads) {
     }
     workspace_.synchronize();
     if (status != GGML_STATUS_SUCCESS) {
-        last_compute_status_ = status;
         LOG_ERROR("%s compute failed: %s", get_desc().c_str(), ggml_status_to_string(status));
         return false;
     }
@@ -824,7 +836,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
     const auto& cached_plan     = resolve_graph_cut_plan(graph);
     const auto full_measurement = measure(graph, cached_plan.compute_buffer_size);
     if (full_measurement.buffers.empty()) {
-        last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
         return std::nullopt;
     }
     auto manager         = residency_manager.lock();
@@ -895,9 +906,7 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
             SegmentGraphBindings& bindings;
             ggml_context* context;
             ~SegmentCleanup() {
-                if (!runner.workspace_.segment_end()) {
-                    runner.last_compute_status_ = GGML_STATUS_FAILED;
-                }
+                runner.workspace_.segment_end();
                 bindings.restore();
                 weights.segment_end();
                 ggml_free(context);
@@ -907,7 +916,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
 
         auto measurement = segmented ? measure(segment_graph, segment.compute_buffer_size) : full_measurement;
         if (!workspace_.prepare(measurement)) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
             return fail_segment("workspace preparation");
         }
         const size_t cut_bytes       = last ? 0 : cut_cache_.estimate_output_bytes(graph, segment);
@@ -922,11 +930,7 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
                 sync_runtime_residency();
                 requests = memory_requests(measurement.buffers, new_cache_bytes);
             }
-            const bool ready = weights.ensure_segment_capacity(index, requests);
-            if (!ready && manager != nullptr) {
-                last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
-            }
-            return ready;
+            return weights.ensure_segment_capacity(index, requests);
         };
         if (!weights.segment_start(index, ensure_capacity)) {
             return fail_segment("weight preparation");
@@ -935,17 +939,12 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
         if (!workspace_.measurement_matches(segment_graph, measurement)) {
             measurement = measure(segment_graph, segment.compute_buffer_size);
         }
-        if (!workspace_.prepare(measurement)) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
-            return fail_segment("workspace preparation");
-        }
-        if (!ensure_capacity()) {
+        if (!workspace_.prepare(measurement) || !ensure_capacity()) {
             return fail_segment("workspace capacity check");
         }
         if (!workspace_.allocate(segment_graph, [&](ggml_backend_sched_t scheduler, ggml_cgraph* current) {
                 pin_multi_device_nodes(scheduler, current);
             })) {
-            last_compute_status_ = GGML_STATUS_ALLOC_FAILED;
             return fail_segment("workspace allocation");
         }
         for (const auto& size : measurement.buffers) {
@@ -983,7 +982,6 @@ std::optional<Tensor<float>> GGMLRunner::execute_graph(ggml_cgraph* graph, int n
             }
         }
         if (!workspace_.segment_end()) {
-            last_compute_status_ = GGML_STATUS_FAILED;
             return fail_segment("workspace synchronization");
         }
         // Final outputs and their callbacks may still be views of consumed cuts.

src/core/ggml_runner.h

@@ -69,6 +69,8 @@ struct GGMLRunnerContext {
     ggml_context* ggml_ctx                                           = nullptr;
     bool flash_attn_enabled                                          = false;
     bool sage_attn_enabled                                           = false;
+    bool sol_attn_enabled                                            = false;
+    float sol_attn_tau                                               = 1.f;
     float linear_scale                                               = 0.f;
     float attn_scale                                                 = 0.f;
     bool conv2d_direct_enabled                                       = false;
@@ -130,8 +132,7 @@ ggml_tensor* ggml_ext_attention_ext(GGMLRunnerContext* ctx,
 struct GGMLRunner {
 private:
     std::map<ggml_backend_t, size_t> logged_compute_bytes_;
-    size_t logged_segment_count_     = 0;
-    ggml_status last_compute_status_ = GGML_STATUS_SUCCESS;
+    size_t logged_segment_count_ = 0;
 
     sd::ComputeWorkspace::Measurement measure(ggml_cgraph* graph, size_t direct_bytes);
     std::vector<DeviceMemoryRequest> memory_requests(const std::vector<sd::BackendBufferSize>& sizes,
@@ -179,6 +180,9 @@ protected:
 
     bool flash_attn_enabled    = false;
     bool sage_attn_enabled     = false;
+    bool sol_attn_enabled      = false;
+    float sol_attn_tau         = 1.f;
+    bool sol_attn_graph_logged = false;
     float linear_scale         = 0.f;
     float attn_scale           = 0.f;
     bool conv2d_direct_enabled = false;
@@ -336,8 +340,6 @@ public:
                                              bool no_return                            = false,
                                              const std::function<bool()>& read_outputs = {});
 
-    ggml_status last_compute_status() const { return last_compute_status_; }
-
     void set_flash_attention_enabled(bool enabled) {
         flash_attn_enabled = enabled;
     }
@@ -350,6 +352,16 @@ public:
         }
     }
 
+    void set_sol_attention_enabled(bool enabled, float tau) {
+        if (sol_attn_enabled != enabled || sol_attn_tau != tau) {
+            free_cache_ctx_and_buffer();
+            graph_cut_plan_cache_.graph_cut_plans.clear();
+            sol_attn_enabled      = enabled;
+            sol_attn_tau          = tau;
+            sol_attn_graph_logged = false;
+        }
+    }
+
     void set_scale_overrides(float linear_scale, float attn_scale) {
         this->linear_scale = linear_scale;
         this->attn_scale   = attn_scale;

src/model.h

@@ -252,14 +252,6 @@ static inline bool sd_version_is_sensenova_u1(SDVersion version) {
     return version == VERSION_SENSENOVA_U1_5;
 }
 
-static inline bool sd_version_supports_video_generation(SDVersion version) {
-    return version == VERSION_SVD || sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_lingbot_video(version) || sd_version_is_ltxav(version) || sd_version_is_minimax_h3(version);
-}
-
-static inline bool sd_version_supports_image_generation(SDVersion version) {
-    return !sd_version_supports_video_generation(version);
-}
-
 static inline bool sd_version_uses_flux_vae(SDVersion version) {
     if (sd_version_is_flux(version) || sd_version_is_z_image(version) || sd_version_is_boogu_image(version) || sd_version_is_longcat(version)) {
         return true;

src/model_manager.cpp

@@ -1613,8 +1613,7 @@ void ModelManager::remove_runtime_owner(uintptr_t owner_id) {
 
 ModelManager::CapacityCheck ModelManager::check_capacity(
     const DeviceMemoryRequest& request,
-    const std::vector<TensorState*>& states,
-    bool log_details) const {
+    const std::vector<TensorState*>& states) const {
     CapacityCheck result;
     if (request.compute_backend == nullptr || sd_backend_is_cpu(request.compute_backend)) {
         return result;
@@ -1632,23 +1631,16 @@ ModelManager::CapacityCheck ModelManager::check_capacity(
         }
         size_t free_bytes = 0, total_bytes = 0;
         ggml_backend_dev_memory(device, &free_bytes, &total_bytes);
-        const size_t weights_resident = compute_backend_resident_bytes(backend);
-        const size_t other_runtime    = other_runtime_resident_bytes(request.owner_id, backend);
-        const size_t resident         = add(weights_resident, add(other_runtime, request.runtime_resident_bytes));
-        if (log_details) {
-            LOG_WARN("model manager memory on %s: reported free %.2f MB / total %.2f MB, tracked weights %.2f MB / other runtime %.2f MB / current runtime %.2f MB",
-                        ggml_backend_name(backend),
-                        free_bytes / (1024.0 * 1024.0), total_bytes / (1024.0 * 1024.0),
-                        weights_resident / (1024.0 * 1024.0), other_runtime / (1024.0 * 1024.0),
-                        request.runtime_resident_bytes / (1024.0 * 1024.0));
-        }
         if (free_bytes == 0 && total_bytes == 0) {
             return SIZE_MAX;
         }
         // Vulkan's heap budget subtraction can underflow when usage exceeds the budget.
-        if (total_bytes > 0 && free_bytes > total_bytes && sd_backend_is(backend, "Vulkan")) {
+        if (total_bytes > 0 && free_bytes > total_bytes) {
             return size_t{0};
         }
+        const size_t resident = add(compute_backend_resident_bytes(backend),
+                                       add(other_runtime_resident_bytes(request.owner_id, backend),
+                                           request.runtime_resident_bytes));
         if (total_bytes > 0) {
             free_bytes = std::min(free_bytes, resident < total_bytes ? total_bytes - resident : 0);
         }
@@ -1794,7 +1786,7 @@ bool ModelManager::ensure_compute_backend_capacity(
         }
     }
 
-    const auto capacity                = check_capacity(request, required_states, true);
+    const auto capacity                = check_capacity(request, required_states);
     const std::string available_device = capacity.available_device_bytes == SIZE_MAX
                                              ? "unknown"
                                              : sd_format("%.2f MB", capacity.available_device_bytes / (1024.0 * 1024.0));

src/model_manager.h

@@ -157,8 +157,7 @@ private:
         }
     };
     CapacityCheck check_capacity(const DeviceMemoryRequest& request,
-                                 const std::vector<TensorState*>& states,
-                                 bool log_details = false) const;
+                                 const std::vector<TensorState*>& states) const;
 
     ggml_backend_buffer_type_t params_buffer_type_for(const TensorState& state) const;
     ggml_backend_buffer_type_t split_buffer_type_for(const TensorState& state) const;

src/pipeline/diffusion_engine.cpp

@@ -899,7 +899,60 @@ bool StableDiffusionGGML::set_sage_attention_enabled(bool enabled) {
     return true;
 }
 
+bool StableDiffusionGGML::set_sol_attention_enabled(bool enabled, float tau) {
+    if (!diffusion_model || !std::isfinite(tau)) {
+        LOG_ERROR("Sol-Attn requires a diffusion model and finite tau");
+        return false;
+    }
+    if (enabled) {
+        if (config_->params.sage_attn) {
+            LOG_ERROR("Sol-Attn and SageAttention cannot be enabled together");
+            return false;
+        }
+#ifndef SD_USE_UPSTREAM_GGML
+        auto* ctx = ggml_init({4 * ggml_tensor_overhead(), nullptr, true});
+        if (!ctx) {
+            return false;
+        }
+        auto* q        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* k        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* v        = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, 128, 128, 1, 1);
+        auto* op       = ggml_sol_attn(ctx, q, k, v, 1.f / sqrtf(128.f), tau);
+        bool supported = true;
+        for (auto backend : backend_manager.runtime_backends(SDBackendModule::DIFFUSION)) {
+            if (!ggml_backend_supports_op(backend, op)) {
+                LOG_ERROR("Sol-Attn is unavailable on %s; it requires patched GGML, CUDA 12.0 or newer, and SM80 or newer kernels", ggml_backend_name(backend));
+                supported = false;
+            }
+        }
+        ggml_free(ctx);
+        if (!supported) {
+            return false;
+        }
+#else
+        LOG_ERROR("Sol-Attn requires -DSD_USE_UPSTREAM_GGML=OFF and a CUDA backend");
+        return false;
+#endif
+    }
+    diffusion_model->set_sol_attention_enabled(enabled, tau);
+    if (high_noise_diffusion_model) {
+        high_noise_diffusion_model->set_sol_attention_enabled(enabled, tau);
+    }
+    if (enabled) {
+        LOG_INFO("Using Sol-Attn (tau=%g, diagonal threshold) in diffusion; unsupported attention uses flash/default attention", tau);
+    }
+    return true;
+}
+
 bool StableDiffusionGGML::init(const sd_ctx_params_t* sd_ctx_params) {
+    if (sd_ctx_params->sol_attn && sd_ctx_params->sage_attn) {
+        LOG_ERROR("Sol-Attn and SageAttention cannot be enabled together");
+        return false;
+    }
+    if (!std::isfinite(sd_ctx_params->sol_attn_tau)) {
+        LOG_ERROR("Sol-Attn tau must be finite");
+        return false;
+    }
 #ifdef SD_USE_UPSTREAM_GGML
     LOG_WARN(
         "Using upstream GGML: INT8 tensorwise/convrot is disabled and FP8 weights are "
@@ -1180,6 +1233,9 @@ bool StableDiffusionGGML::validate_and_load_runners() {
     if (sd_ctx_params->sage_attn && !set_sage_attention_enabled(true)) {
         return false;
     }
+    if (sd_ctx_params->sol_attn && !set_sol_attention_enabled(true, sd_ctx_params->sol_attn_tau)) {
+        return false;
+    }
     LOG_VERBOSE("validating model metadata");
 
     std::set<std::string> ignore_tensors;
@@ -2766,8 +2822,7 @@ sd::Tensor<float> StableDiffusionGGML::decode_first_stage(const sd::Tensor<float
     auto decoded                      = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
     const bool prefer_temporal_tiling = decode_video && first_stage_model->can_temporal_tile_decode();
     while (decoded.empty() &&
-           sd::backend_fit::prepare_vae_decode_retry_tiling(vae_tiling_params, prefer_temporal_tiling,
-                                                            first_stage_model->last_compute_status())) {
+           sd::backend_fit::prepare_vae_decode_retry_tiling(vae_tiling_params, prefer_temporal_tiling)) {
         decoded = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
     }
     return decoded;

src/pipeline/diffusion_engine.h

@@ -313,6 +313,7 @@ public:
 
     bool init(const sd_ctx_params_t* sd_ctx_params);
     bool set_sage_attention_enabled(bool enabled);
+    bool set_sol_attention_enabled(bool enabled, float tau);
 
     bool uses_tae() const;
 

src/pipeline/image.cpp

@@ -789,9 +789,15 @@ namespace sd::pipeline {
             return false;
         }
 
-        if (!sd_version_supports_image_generation(sd->version)) {
-            LOG_ERROR("%s cannot be run with generate_image(); use generate_video() or --mode vid_gen in the CLI",
-                      model_version_to_str[sd->version]);
+        // MiniMax-H3 is video-only. Its denoiser always splits the packed latent into a video and an
+        // audio half, and only generate_video ever computes the audio length, so reaching this
+        // function with an H3 checkpoint is guaranteed to die on
+        // GGML_ASSERT(!audio_input_cache.empty()) with a core dump, after the several minutes it
+        // takes to load the weights, and with nothing in the output pointing at the missing --mode.
+        // (The AnimateDiff path below routes vid_gen back through here, but that is SD1.5 plus a
+        // motion module, never H3.)
+        if (sd_version_is_minimax_h3(sd->version)) {
+            LOG_ERROR("MiniMax-H3 is a video model and cannot be run in img_gen mode; use --mode vid_gen");
             return false;
         }
 

src/pipeline/video.cpp

@@ -470,15 +470,11 @@ namespace sd::pipeline {
         sd::Tensor<float> end_image;
 
         if (sd_vid_gen_params->init_image.data) {
-            start_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height),
-                sd->get_image_channels());
+            start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, request->width, request->height);
         }
 
         if (sd_vid_gen_params->end_image.data) {
-            end_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height),
-                sd->get_image_channels());
+            end_image = sd_image_to_tensor(sd_vid_gen_params->end_image, request->width, request->height);
         }
 
         if (sd_version_is_minimax_h3(sd->version)) {
@@ -1420,9 +1416,7 @@ namespace sd::pipeline {
         sd::Tensor<float> video_mask = make_ltxav_video_denoise_mask(video_latent, 1.f);
 
         if (sd_vid_gen_params->init_image.data != nullptr) {
-            sd::Tensor<float> start_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height),
-                sd->get_image_channels());
+            sd::Tensor<float> start_image = sd_image_to_tensor(sd_vid_gen_params->init_image, image_width, image_height);
             if (!apply_ltxav_condition_image_by_latent_index(sd,
                                                              start_image,
                                                              &video_latent,
@@ -1435,9 +1429,7 @@ namespace sd::pipeline {
         }
 
         if (sd_vid_gen_params->end_image.data != nullptr) {
-            sd::Tensor<float> end_image = ensure_image_tensor_channels(
-                sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height),
-                sd->get_image_channels());
+            sd::Tensor<float> end_image        = sd_image_to_tensor(sd_vid_gen_params->end_image, image_width, image_height);
             sd::Tensor<float> end_image_latent = encode_ltxav_condition_image(sd, end_image, "end");
             if (end_image_latent.empty()) {
                 return false;

src/stable-diffusion.cpp

@@ -338,6 +338,8 @@ void sd_ctx_params_init(sd_ctx_params_t* sd_ctx_params) {
     sd_ctx_params->enable_mmap               = false;
     sd_ctx_params->diffusion_flash_attn      = false;
     sd_ctx_params->sage_attn                 = false;
+    sd_ctx_params->sol_attn                  = false;
+    sd_ctx_params->sol_attn_tau              = 1.f;
     sd_ctx_params->linear_scale              = 0.f;
     sd_ctx_params->attn_scale                = 0.f;
     sd_ctx_params->vae_format                = SD_VAE_FORMAT_AUTO;
@@ -394,6 +396,8 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              "flash_attn: %s\n"
              "diffusion_flash_attn: %s\n"
              "sage_attn: %s\n"
+             "sol_attn: %s\n"
+             "sol_attn_tau: %g\n"
              "linear_scale: %g\n"
              "attn_scale: %g\n"
              "vae_format: %s\n",
@@ -434,6 +438,8 @@ char* sd_ctx_params_to_str(const sd_ctx_params_t* sd_ctx_params) {
              BOOL_STR(sd_ctx_params->flash_attn),
              BOOL_STR(sd_ctx_params->diffusion_flash_attn),
              BOOL_STR(sd_ctx_params->sage_attn),
+             BOOL_STR(sd_ctx_params->sol_attn),
+             sd_ctx_params->sol_attn_tau,
              sd_ctx_params->linear_scale,
              sd_ctx_params->attn_scale,
              sd_vae_format_name(sd_ctx_params->vae_format));
@@ -630,6 +636,14 @@ struct sd_ctx_t {
     StableDiffusionGGML* sd = nullptr;
 };
 
+static bool sd_version_supports_video_generation(SDVersion version) {
+    return version == VERSION_SVD || sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_lingbot_video(version) || sd_version_is_ltxav(version) || sd_version_is_minimax_h3(version);
+}
+
+static bool sd_version_supports_image_generation(SDVersion version) {
+    return !sd_version_supports_video_generation(version);
+}
+
 sd_ctx_t* new_sd_ctx(const sd_ctx_params_t* sd_ctx_params) {
     sd_ctx_t* sd_ctx = (sd_ctx_t*)malloc(sizeof(sd_ctx_t));
     if (sd_ctx == nullptr) {