diff --git a/docs/edit.md b/docs/edit.md
index 9f8a9c3f..0bd3b562 100644
--- a/docs/edit.md
+++ b/docs/edit.md
@@ -26,6 +26,9 @@ Stable-diffusion.spp also supports basic Unet-based editing models like instruct
## Configuring Reference Modes (`--ref-image-args`)
+For a one-time input transform before reference presets and model processing,
+including cropping, padding, and resizing algorithms, see [Image preprocessing](./image_preprocessing.md).
+
Different DiT-based editing models require different configurations to process reference images correctly (e.g., whether to use a Vision Language Model (VLM) encoder or pass VAE-encoded images directly to the DiT).
To simplify this, we provide **Presets**. By default, the system automatically selects the best preset based on the model architecture. However, you can override this using the `--ref-image-args` argument.
diff --git a/docs/image_preprocessing.md b/docs/image_preprocessing.md
new file mode 100644
index 00000000..71978175
--- /dev/null
+++ b/docs/image_preprocessing.md
@@ -0,0 +1,173 @@
+# Image preprocessing
+
+Use `--image-preprocess` to transform each image input once, before generation:
+
+```sh
+sd-cli ... \
+ --image-preprocess "target=init,mode=crop-resize,filter=lanczos,antialias=true" \
+ --image-preprocess "target=mask,filter=nearest-exact" \
+ --image-preprocess "target=ref,index=0,mode=fit-pad,width=768,height=768,filter=bicubic"
+```
+
+CLI and server image loaders decode at the original resolution. The generation
+entry point merges input defaults with user rules and prepares one transformed
+image per input. The original pipeline then consumes those images, including
+its mandatory canvas adaptation, reference resizing, and encoder preprocessing.
+
+```text
+native-resolution image
+ -> input defaults + user overrides
+ -> one input transform
+ -> original generation pipeline and model-specific processing
+```
+
+These rules do not override internal VAE, CLIP/VLM, ControlNet, or pixel-patch preprocessing.
+`--ref-image-args` retains its existing meaning and runs after this input transform.
+
+## Inputs and defaults
+
+| `target` | Input | Default geometry | Indexed? |
+| --- | --- | --- | --- |
+| `init` | img2img image or video first frame | Center crop to the generation aspect ratio, then resize | No |
+| `end` | Video last frame | Center crop, then resize | No |
+| `mask` | Inpainting mask | Inherit init geometry; otherwise center crop, then resize | No |
+| `control` | Control image | Center crop, then resize | No |
+| `ref` | Reference images | Preserve source dimensions | Yes |
+| `ip-adapter` | IP-Adapter image | Preserve source dimensions | No |
+| `id` | PhotoMaker identity images | Preserve source dimensions | Yes |
+| `control-frame` | Control video frames | Center crop, then resize | Yes |
+
+Canvas defaults use the aligned generation dimensions. Reference, IP-Adapter,
+and identity inputs use their original dimensions unless overridden. Default
+resampling is nearest for images and nearest-exact for masks.
+
+These defaults are shared by CLI, server, and C API. Moving geometry out of
+the loaders replaces the previous CLI/server BOX/sRGB resizing, so default
+pixels are not guaranteed to match earlier builds.
+
+Reference video and audio preprocessing are outside these image rules.
+Preprocessing options apply to `img_gen` and `vid_gen`, not standalone upscale
+or ADetailer mode. ADetailer clears the user's rules for its internal crops.
+
+## Rules
+
+Rules are comma-separated `key=value` lists. Repeat the CLI option or separate
+rules with semicolons. Every rule requires a `target` and at least one option.
+Rule syntax and input compatibility are checked when image/video generation
+starts. Unknown keys, invalid values, duplicate keys in a rule, missing images,
+and out-of-range indices cause generation to fail with an error log.
+
+Omit `index` to configure every image of that type; otherwise use a zero-based
+index. CLI directory inputs follow filename order. Indexed rules override
+type-wide rules field by field, regardless of order. At equal specificity,
+the last value for a field wins. `auto` selects the input preset.
+
+| `mode` | Input transform |
+| --- | --- |
+| `auto` | Use the input's default geometry |
+| `none` | Keep source dimensions without resizing, cropping, or padding |
+| `stretch` | Resize to the target dimensions |
+| `crop` | Crop a target-sized rectangle without resizing; fail if the source is too small |
+| `crop-resize` | Crop to the target aspect ratio, then resize |
+| `fit-pad` | Fit the entire image inside the target dimensions, preserving aspect ratio, then pad |
+
+`width` and `height` must be specified together as positive integers. They
+override the input transform's dimensions, not the generation or encoder size.
+For a native-size preset, specifying dimensions without a mode selects stretch.
+`mode=none` with explicit dimensions different from the source is contradictory
+and is rejected.
+
+`anchor=center|top|bottom|left|right` selects crop/padding placement.
+`pad_color=#RRGGBB` or `#RRGGBBAA` selects padding, defaulting to opaque black.
+A grayscale mask uses the first color component.
+
+`filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos` selects resampling.
+`antialias=auto|true|false` enables antialiasing automatically for filtered
+downscaling; explicit true requires bilinear, bicubic, or Lanczos.
+Filtered RGBA resizing uses premultiplied alpha.
+
+`canny=true|false` enables edge detection for any supported image target,
+defaulting to `false`. It runs once after geometry, before the original
+generation pipeline, including with `mode=none`. Grayscale, grayscale-alpha,
+RGB, and RGBA inputs are supported; alpha is preserved.
+
+Each input has its own Canny setting. Indexed rules can enable or disable it
+for individual references, identity images, or video control frames.
+
+```sh
+--image-preprocess "target=init,mode=fit-pad,canny=true"
+--image-preprocess "target=ref,index=0,mode=none,canny=true"
+--image-preprocess "target=control-frame,index=2,canny=true"
+```
+
+Init and mask sources must have the same dimensions. The mask inherits the
+init crop, resize, and padding coordinates, while retaining its own filter,
+padding value, and Canny setting. Conflicting mask geometry is rejected. An
+omitted mask remains absent until the original pipeline creates its default mask.
+
+## Downstream behavior
+
+`mode=none` only skips the input geometry transform. For example:
+
+```sh
+--image-preprocess "target=init,mode=none" \
+--image-preprocess "target=ref,mode=none"
+```
+
+The init image is still adapted to the generation canvas by the original
+pipeline. Reference images still follow `--ref-image-args` and model-specific
+resizing. CLIP retains its fixed input dimensions and normalization. HiDream-O1
+retains its original pixel-reference and visual preprocessing.
+
+Existing sharing between consumers is preserved: for example, Wan img2video
+uses the same adapted first frame for VAE conditioning and CLIP. High-resolution
+passes reuse the prepared images and apply their original size adaptation;
+they do not apply the user's crop a second time.
+
+To disable reference resizing before VAE encoding, use
+`--ref-image-args "resize_before_vae=false"` or the server field
+`"ref_image_args": "resize_before_vae=false"`. This is separate from
+`target=ref,mode=none`, which only skips input geometry. Model constraints
+still apply.
+
+## Server requests
+
+Native image/video requests and SDAPI accept `image_preprocess` as a string or
+an array of rule strings:
+
+```json
+{
+ "image_preprocess": [
+ "target=init,mode=fit-pad,filter=bicubic",
+ "target=mask,filter=nearest-exact",
+ "target=ref,index=0,mode=none"
+ ]
+}
+```
+
+OpenAI-compatible requests accept it through
+`{...}` in the prompt.
+Request rules replace server-default rules. Generation metadata records the
+user rules; image encodings and channel conventions are unchanged.
+
+## C API
+
+Set `image_preprocess` on the existing image/video generation parameters.
+The `generate_image()` and `generate_video()` signatures are unchanged:
+
+```c
+sd_img_gen_params_t params;
+sd_img_gen_params_init(¶ms);
+/* Set prompt, original-resolution input images, and generation options. */
+params.image_preprocess.rules = "target=init,mode=crop-resize,filter=lanczos;"
+ "target=mask,filter=nearest-exact";
+bool ok = generate_image(ctx, ¶ms, &images, &count);
+```
+
+Both generation parameter initializers set `image_preprocess.rules` to `NULL`,
+selecting input presets. Rule strings are borrowed for the synchronous call.
+The library owns temporary transformed pixels; caller images and arrays are
+not modified. Add `canny=true` to the desired target's rule in
+`image_preprocess.rules` to enable Canny.
+
+The parameter structs have grown; applications and bindings must be rebuilt.
diff --git a/examples/cli/README.md b/examples/cli/README.md
index fe47122b..63df3e91 100644
--- a/examples/cli/README.md
+++ b/examples/cli/README.md
@@ -14,6 +14,12 @@ equivalent to `--log-level verbose`. If repeated, the last logging option wins.
For direct image repair or automatic post-generation YOLOv8 detection followed by cropped inpainting, see
[ADetailer](../../docs/adetailer.md).
+Use repeatable `--image-preprocess` rules to select resizing, cropping, padding,
+and resampling separately for each image input. Add `canny=true` to any input
+rule for edge detection. See
+[Image preprocessing](../../docs/image_preprocessing.md) for input selectors,
+input defaults, downstream model processing, mask alignment, and examples.
+
Metadata mode inspects PNG/JPEG container metadata without loading any model:
```bash
diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp
index 1a0bb28e..a2791dda 100644
--- a/examples/cli/main.cpp
+++ b/examples/cli/main.cpp
@@ -41,7 +41,6 @@ struct SDCliParams {
std::string metadata_format = "text";
sd_log_level_t log_level = SD_LOG_INFO;
- bool canny_preprocess = false;
bool convert_name = false;
preview_t preview_method = PREVIEW_NONE;
@@ -107,10 +106,6 @@ struct SDCliParams {
};
options.bool_options = {
- {"",
- "--canny",
- "apply canny preprocessor (edge detection)",
- true, &canny_preprocess},
{"",
"--convert-name",
"convert tensor name (for convert mode)",
@@ -268,7 +263,6 @@ struct SDCliParams {
<< " metadata_format: \"" << metadata_format << "\",\n"
<< " log_level: " << log_level_name(log_level) << ",\n"
<< " color: " << (color ? "true" : "false") << ",\n"
- << " canny_preprocess: " << (canny_preprocess ? "true" : "false") << ",\n"
<< " convert_name: " << (convert_name ? "true" : "false") << ",\n"
<< " preview_method: " << previews_str[preview_method] << ",\n"
<< " preview_interval: " << preview_interval << ",\n"
@@ -328,9 +322,7 @@ void sd_log_cb(enum sd_log_level_t level, const char* log, void* data) {
bool load_images_from_dir(const std::string dir,
std::vector& images,
- int expected_width = 0,
- int expected_height = 0,
- int max_image_num = 0) {
+ int max_image_num = 0) {
if (!fs::exists(dir) || !fs::is_directory(dir)) {
LOG_ERROR("'%s' is not a valid directory\n", dir.c_str());
return false;
@@ -358,7 +350,7 @@ bool load_images_from_dir(const std::string dir,
int width = 0;
int height = 0;
int loaded_channel = 0;
- uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, expected_width, expected_height);
+ uint8_t* image_buffer = load_image_from_file(path.c_str(), width, height, loaded_channel, 0, 0);
if (image_buffer == nullptr) {
LOG_ERROR("load image from '%s' failed", path.c_str());
return false;
@@ -654,8 +646,8 @@ int main(int argc, const char* argv[]) {
SDContextParams ctx_params;
SDGenerationParams gen_params;
- parse_args(argc, argv, cli_params, ctx_params, gen_params);
sd_set_log_callback(sd_log_cb, (void*)&cli_params);
+ parse_args(argc, argv, cli_params, ctx_params, gen_params);
if (cli_params.mode == METADATA) {
MetadataReadOptions options;
@@ -751,16 +743,8 @@ int main(int argc, const char* argv[]) {
auto load_image_and_update_size = [&](const std::string& path,
SDImageOwner& image,
- bool resize_image = true,
int expected_channel = 3) -> bool {
- int expected_width = 0;
- int expected_height = 0;
- if (resize_image && gen_params.width_and_height_are_set()) {
- expected_width = gen_params.width;
- expected_height = gen_params.height;
- }
-
- if (!load_sd_image_from_file(image.put(), path.c_str(), expected_width, expected_height, expected_channel)) {
+ if (!load_sd_image_from_file(image.put(), path.c_str(), 0, 0, expected_channel)) {
LOG_ERROR("load image from '%s' failed", path.c_str());
return false;
}
@@ -783,7 +767,7 @@ int main(int argc, const char* argv[]) {
if (gen_params.init_image_path.size() > 0) {
const bool native_init = cli_params.mode == IMG_GEN || cli_params.mode == ADETAILER;
- if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, true, native_init ? 0 : 3)) {
+ if (!load_image_and_update_size(gen_params.init_image_path, gen_params.init_image, native_init ? 0 : 3)) {
return 1;
}
}
@@ -798,7 +782,7 @@ int main(int argc, const char* argv[]) {
gen_params.ref_images.clear();
for (auto& path : gen_params.ref_image_paths) {
SDImageOwner ref_image({0, 0, 0, nullptr});
- if (!load_image_and_update_size(path, ref_image, false, 0)) {
+ if (!load_image_and_update_size(path, ref_image, 0)) {
return 1;
}
gen_params.ref_images.push_back(std::move(ref_image));
@@ -839,41 +823,22 @@ int main(int argc, const char* argv[]) {
if (gen_params.mask_image_path.size() > 0) {
if (!load_sd_image_from_file(gen_params.mask_image.put(),
gen_params.mask_image_path.c_str(),
- gen_params.get_resolved_width(),
- gen_params.get_resolved_height(),
+ 0,
+ 0,
1)) {
LOG_ERROR("load image from '%s' failed", gen_params.mask_image_path.c_str());
return 1;
}
- } else {
- sd_image_t generated_mask = {0, 0, 1, nullptr};
- generated_mask.data = (uint8_t*)malloc(gen_params.get_resolved_width() * gen_params.get_resolved_height());
- if (generated_mask.data == nullptr) {
- LOG_ERROR("malloc mask image failed");
- return 1;
- }
- generated_mask.width = gen_params.get_resolved_width();
- generated_mask.height = gen_params.get_resolved_height();
- memset(generated_mask.data, 255, gen_params.get_resolved_width() * gen_params.get_resolved_height());
- gen_params.mask_image.reset(generated_mask);
}
if (gen_params.control_image_path.size() > 0) {
if (!load_sd_image_from_file(gen_params.control_image.put(),
gen_params.control_image_path.c_str(),
- gen_params.get_resolved_width(),
- gen_params.get_resolved_height())) {
+ 0,
+ 0)) {
LOG_ERROR("load image from '%s' failed", gen_params.control_image_path.c_str());
return 1;
}
- if (cli_params.canny_preprocess) { // apply preprocessor
- preprocess_canny(gen_params.control_image.get(),
- 0.08f,
- 0.08f,
- 0.8f,
- 1.0f,
- false);
- }
}
if (gen_params.ip_adapter_image_path.size() > 0) {
@@ -890,8 +855,6 @@ int main(int argc, const char* argv[]) {
gen_params.control_frames.clear();
if (!load_images_from_dir(gen_params.control_video_path,
gen_params.control_frames,
- gen_params.get_resolved_width(),
- gen_params.get_resolved_height(),
gen_params.video_frames)) {
return 1;
}
@@ -900,10 +863,7 @@ int main(int argc, const char* argv[]) {
if (!gen_params.pm_id_images_dir.empty()) {
gen_params.pm_id_images.clear();
if (!load_images_from_dir(gen_params.pm_id_images_dir,
- gen_params.pm_id_images,
- 0,
- 0,
- 0)) {
+ gen_params.pm_id_images)) {
return 1;
}
}
diff --git a/examples/common/common.cpp b/examples/common/common.cpp
index 25f78d3b..1eadc17c 100644
--- a/examples/common/common.cpp
+++ b/examples/common/common.cpp
@@ -1128,6 +1128,9 @@ ArgOptions SDGenerationParams::get_options() {
"Key-value list to set up the way the reference images are processed (empty = auto-detect from model weigths)",
(int)',',
&ref_image_args},
+ {"", "--image-preprocess",
+ "Image preprocessing rule: target=init|end|mask|control|ref|ip-adapter|id|control-frame,index=N,mode=auto|none|stretch|crop|crop-resize|fit-pad,filter=auto|nearest|nearest-exact|bilinear|bicubic|lanczos,antialias=auto|true|false,width=W,height=H,anchor=center|top|bottom|left|right,pad_color=#RRGGBB[AA],canny=true|false. Repeat for multiple rules.",
+ (int)';', &image_preprocess},
};
options.int_options = {
@@ -1308,11 +1311,6 @@ ArgOptions SDGenerationParams::get_options() {
"automatically increase the indices of references images based on the order they are listed (starting with 1).",
true,
&increase_ref_index},
- {"",
- "--disable-auto-resize-ref-image",
- "disable auto resize of ref images",
- false,
- &auto_resize_ref_image},
{"",
"--circular",
"enable circular padding on both axes for tileable output",
@@ -1870,8 +1868,6 @@ bool decode_base64_image(const std::string& encoded_input,
static bool parse_image_json_field(const json& parent,
const char* key,
int channels,
- int expected_width,
- int expected_height,
SDImageOwner& out_image) {
if (!parent.contains(key)) {
return true;
@@ -1883,14 +1879,12 @@ static bool parse_image_json_field(const json& parent,
if (!parent.at(key).is_string()) {
return false;
}
- return decode_base64_image(parent.at(key).get(), channels, expected_width, expected_height, out_image);
+ return decode_base64_image(parent.at(key).get(), channels, 0, 0, out_image);
}
static bool parse_image_array_json_field(const json& parent,
const char* key,
int channels,
- int expected_width,
- int expected_height,
std::vector& out_images) {
if (!parent.contains(key)) {
return true;
@@ -1909,7 +1903,7 @@ static bool parse_image_array_json_field(const json& parent,
return false;
}
SDImageOwner image;
- if (!decode_base64_image(item.get(), channels, expected_width, expected_height, image)) {
+ if (!decode_base64_image(item.get(), channels, 0, 0, image)) {
return false;
}
out_images.push_back(std::move(image));
@@ -2008,6 +2002,29 @@ static bool resolve_model_file_from_dir(const std::string& model_name,
return false;
}
+bool SDGenerationParams::parse_image_preprocess_json(const std::string& json_str) {
+ const auto value = json::parse(json_str, nullptr, false);
+ std::string rules;
+ if (value.is_string()) {
+ rules = value.get();
+ } else if (value.is_array()) {
+ for (const auto& item : value) {
+ if (!item.is_string()) {
+ LOG_ERROR("image_preprocess must contain rule strings");
+ return false;
+ }
+ if (!rules.empty())
+ rules += ";";
+ rules += item.get();
+ }
+ } else {
+ LOG_ERROR("image_preprocess must be a string or array of strings");
+ return false;
+ }
+ image_preprocess = std::move(rules);
+ return true;
+}
+
bool SDGenerationParams::from_json_str(
const std::string& json_str,
const std::function& lora_path_resolver) {
@@ -2019,6 +2036,9 @@ bool SDGenerationParams::from_json_str(
return false;
}
+ if (j.contains("image_preprocess") && !parse_image_preprocess_json(j["image_preprocess"].dump()))
+ return false;
+
auto load_if_exists = [&](const char* key, auto& out) {
if (j.contains(key)) {
using T = std::decay_t;
@@ -2056,6 +2076,7 @@ bool SDGenerationParams::from_json_str(
load_if_exists("cache_mode", cache_mode);
load_if_exists("cache_option", cache_option);
load_if_exists("scm_mask", scm_mask);
+ load_if_exists("ref_image_args", ref_image_args);
load_if_exists("clip_skip", clip_skip);
load_if_exists("width", width);
@@ -2073,7 +2094,6 @@ bool SDGenerationParams::from_json_str(
load_if_exists("moe_boundary", moe_boundary);
load_if_exists("vace_strength", vace_strength);
- load_if_exists("auto_resize_ref_image", auto_resize_ref_image);
load_if_exists("increase_ref_index", increase_ref_index);
load_if_exists("embed_image_metadata", embed_image_metadata);
@@ -2217,37 +2237,23 @@ bool SDGenerationParams::from_json_str(
LOG_ERROR("invalid lora");
return false;
}
- if (!parse_image_json_field(j, "init_image", 0, width, height, init_image)) {
- LOG_ERROR("invalid init_image");
+ auto load_image = [&](const char* key, int channels, SDImageOwner& image) {
+ if (!parse_image_json_field(j, key, channels, image)) {
+ LOG_ERROR("invalid %s", key);
+ return false;
+ }
+ return true;
+ };
+ if (!load_image("init_image", 0, init_image) ||
+ !load_image("end_image", 3, end_image) ||
+ !load_image("mask_image", 1, mask_image) ||
+ !load_image("control_image", 3, control_image) ||
+ !load_image("ip_adapter_image", 3, ip_adapter_image)) {
return false;
}
- if (!parse_image_json_field(j, "end_image", 3, width, height, end_image)) {
- LOG_ERROR("invalid end_image");
- return false;
- }
- if (!parse_image_array_json_field(j,
- "ref_images",
- 0,
- auto_resize_ref_image ? width : 0,
- auto_resize_ref_image ? height : 0,
- ref_images)) {
- LOG_ERROR("invalid ref_images");
- return false;
- }
- if (!parse_image_array_json_field(j, "control_frames", 3, width, height, control_frames)) {
- LOG_ERROR("invalid control_frames");
- return false;
- }
- if (!parse_image_json_field(j, "mask_image", 1, width, height, mask_image)) {
- LOG_ERROR("invalid mask_image");
- return false;
- }
- if (!parse_image_json_field(j, "control_image", 3, width, height, control_image)) {
- LOG_ERROR("invalid control_image");
- return false;
- }
- if (!parse_image_json_field(j, "ip_adapter_image", 3, width, height, ip_adapter_image)) {
- LOG_ERROR("invalid ip_adapter_image");
+ if (!parse_image_array_json_field(j, "ref_images", 0, ref_images) ||
+ !parse_image_array_json_field(j, "control_frames", 3, control_frames)) {
+ LOG_ERROR("invalid input image array");
return false;
}
@@ -2491,6 +2497,10 @@ bool SDGenerationParams::resolve(const std::string& lora_model_dir, const std::s
}
bool SDGenerationParams::validate(SDMode mode) {
+ if (!image_preprocess.empty() && mode != IMG_GEN && mode != VID_GEN) {
+ LOG_ERROR("--image-preprocess requires img_gen or vid_gen mode");
+ return false;
+ }
if (batch_count <= 0) {
LOG_ERROR("error: batch_count must be greater than 0");
return false;
@@ -2666,14 +2676,6 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
pulid_id_weight,
};
- if (!auto_resize_ref_image) {
- if (!ref_image_args.empty()) {
- ref_image_args += ",";
- }
- ref_image_args += "resize_before_vae=0";
- LOG_WARN("Notice: --disable-auto-resize-ref-image is deprecated. Use --ref-image-args \"resize_before_vae=off\" instead.");
- }
-
if (increase_ref_index) {
if (!ref_image_args.empty()) {
ref_image_args += ",";
@@ -2721,6 +2723,7 @@ sd_img_gen_params_t SDGenerationParams::to_sd_img_gen_params_t() {
params.hires.custom_sigmas_count = static_cast(hires_custom_sigmas.size());
params.circular_x = circular || circular_x;
params.circular_y = circular || circular_y;
+ params.image_preprocess = {image_preprocess.c_str()};
return params;
}
@@ -2823,6 +2826,7 @@ sd_vid_gen_params_t SDGenerationParams::to_sd_vid_gen_params_t() {
params.hires.custom_sigmas_count = static_cast(hires_custom_sigmas.size());
params.circular_x = circular || circular_x;
params.circular_y = circular || circular_y;
+ params.image_preprocess = {image_preprocess.c_str()};
return params;
}
@@ -2879,7 +2883,8 @@ std::string SDGenerationParams::to_string() const {
<< " ref_video_audio_paths: " << vec_str_to_string(ref_video_audio_paths) << ",\n"
<< " ref_audio_paths: " << vec_str_to_string(ref_audio_paths) << ",\n"
<< " control_video_path: \"" << control_video_path << "\",\n"
- << " auto_resize_ref_image: " << (auto_resize_ref_image ? "true" : "false") << ",\n"
+ << " image_preprocess: " << image_preprocess << ",\n"
+ << " ref_image_args: " << ref_image_args << ",\n"
<< " increase_ref_index: " << (increase_ref_index ? "true" : "false") << ",\n"
<< " pm_id_images_dir: \"" << pm_id_images_dir << "\",\n"
<< " pm_id_embed_path: \"" << pm_id_embed_path << "\",\n"
@@ -3030,12 +3035,13 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
set_json_basename_if_not_empty(models, "control_net", ctx_params.control_net_path);
root["models"] = std::move(models);
- root["clip_skip"] = gen_params.clip_skip;
- root["strength"] = gen_params.strength;
- root["control_strength"] = gen_params.control_strength;
- root["ip_adapter_strength"] = gen_params.ip_adapter_strength;
- root["auto_resize_ref_image"] = gen_params.auto_resize_ref_image;
- root["increase_ref_index"] = gen_params.increase_ref_index;
+ root["clip_skip"] = gen_params.clip_skip;
+ root["strength"] = gen_params.strength;
+ root["control_strength"] = gen_params.control_strength;
+ root["ip_adapter_strength"] = gen_params.ip_adapter_strength;
+ root["ref_image_args"] = gen_params.ref_image_args;
+ root["image_preprocess"] = gen_params.image_preprocess;
+ root["increase_ref_index"] = gen_params.increase_ref_index;
if (mode == VID_GEN) {
root["video"] = {
{"frame_count", gen_params.video_frames},
diff --git a/examples/common/common.h b/examples/common/common.h
index 0e5c50e3..b46308ac 100644
--- a/examples/common/common.h
+++ b/examples/common/common.h
@@ -200,18 +200,17 @@ struct SDGenerationParams {
std::string ad_prompt;
std::string ad_negative_prompt;
std::string extra_ad_args;
- int clip_skip = -1; // <= 0 represents unspecified
- int width = -1;
- int height = -1;
- int batch_count = 1;
- int qwen_image_layers = 3;
- int64_t seed = 42;
- float strength = 0.75f;
- float control_strength = 0.9f;
- float ip_adapter_strength = 1.0f;
- bool auto_resize_ref_image = true;
- bool increase_ref_index = false;
- bool embed_image_metadata = true;
+ int clip_skip = -1; // <= 0 represents unspecified
+ int width = -1;
+ int height = -1;
+ int batch_count = 1;
+ int qwen_image_layers = 3;
+ int64_t seed = 42;
+ float strength = 0.75f;
+ float control_strength = 0.9f;
+ float ip_adapter_strength = 1.0f;
+ bool increase_ref_index = false;
+ bool embed_image_metadata = true;
std::string init_image_path;
std::string end_image_path;
@@ -247,6 +246,7 @@ struct SDGenerationParams {
std::string extra_tiling_args;
std::string ref_image_args;
+ std::string image_preprocess;
std::string pm_id_images_dir;
std::string pm_id_embed_path;
@@ -310,6 +310,7 @@ struct SDGenerationParams {
ArgOptions get_options();
bool from_json_str(const std::string& json_str,
const std::function& lora_path_resolver = {});
+ bool parse_image_preprocess_json(const std::string& json_str);
bool initialize_cache_params();
void extract_and_remove_lora(const std::string& lora_model_dir);
bool width_and_height_are_set() const;
diff --git a/examples/server/api.md b/examples/server/api.md
index 6c4afd7e..c5cae54c 100644
--- a/examples/server/api.md
+++ b/examples/server/api.md
@@ -148,17 +148,17 @@ Native extension fields:
- any `sdcpp API` fields embedded through `sd_cpp_extra_args` inside `prompt`
-Reference image sizing follows `auto_resize_ref_image`, as in the native and SDAPI APIs.
-The server default is `true`; `--disable-auto-resize-ref-image` sets it to `false`.
-When enabled, uploaded references are center-cropped and resized to the request dimensions.
-If `size` is omitted, the first decoded image establishes those dimensions.
-When disabled, each reference retains its original dimensions.
-The init image and mask still use the request dimensions, independently of this option.
+Uploaded images are decoded at their original dimensions. The first decoded
+image establishes the generation dimensions if `size` is omitted. Input
+geometry follows `image_preprocess`: references preserve their dimensions by
+default, while init and mask use the generation canvas preset.
-To override the server default for one request, include this in `prompt`:
+Reference encoding then follows model presets and `ref_image_args`. To skip
+input geometry for references and disable resizing before VAE encoding, include
+this in `prompt`:
```text
-edit this image {"auto_resize_ref_image":false}
+edit this image {"image_preprocess":"target=ref,mode=none","ref_image_args":"resize_before_vae=false"}
```
Response fields:
@@ -539,7 +539,7 @@ LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEH
| Field | Type |
| --- | --- |
| `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
| `increase_ref_index` | `boolean` |
| `control_strength` | `number` |
| `ip_adapter_strength` | `number` |
@@ -666,7 +666,7 @@ Example:
"strength": 0.75,
"seed": -1,
"batch_count": 1,
- "auto_resize_ref_image": true,
+ "ref_image_args": "",
"increase_ref_index": false,
"control_strength": 0.9,
"ip_adapter_strength": 1.0,
@@ -741,6 +741,17 @@ Example:
### Image Encoding Rules
+Native image/video requests and SDAPI accept `image_preprocess` as a rule string
+or array of rule strings. OpenAI-compatible requests can supply it in
+`sd_cpp_extra_args`. See [Image preprocessing](../../docs/image_preprocessing.md)
+for one-time input geometry, native-resolution decoding, mask alignment, and
+`canny=true` for edge detection on any supported image input.
+
+Image generation also accepts `ref_image_args` as a string (for example,
+`"resize_before_vae=false"`) in native and SDAPI requests, or through
+`sd_cpp_extra_args` in OpenAI-compatible requests. It controls downstream
+reference encoding and is independent of input geometry rules.
+
Any image field accepts:
- a raw base64 string, or
@@ -776,7 +787,8 @@ Top-level scalar fields:
| `strength` | `number` |
| `seed` | `integer` |
| `batch_count` | `integer` |
-| `auto_resize_ref_image` | `boolean` |
+| `ref_image_args` | `string` |
+| `image_preprocess` | `string \| array` |
| `increase_ref_index` | `boolean` |
| `control_strength` | `number` |
| `ip_adapter_strength` | `number` |
diff --git a/examples/server/main.cpp b/examples/server/main.cpp
index 1f93e0dc..42c84df6 100644
--- a/examples/server/main.cpp
+++ b/examples/server/main.cpp
@@ -76,9 +76,9 @@ int main(int argc, const char** argv) {
SDSvrParams svr_params;
SDContextParams ctx_params;
SDGenerationParams default_gen_params;
- parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
sd_set_log_callback(sd_log_cb, (void*)&svr_params);
+ parse_args(argc, argv, svr_params, ctx_params, default_gen_params);
LOG_VERBOSE("version: %s", version_string().c_str());
LOG_VERBOSE("%s", sd_get_system_info());
diff --git a/examples/server/routes_openai.cpp b/examples/server/routes_openai.cpp
index 876b500f..af0cdfaa 100644
--- a/examples/server/routes_openai.cpp
+++ b/examples/server/routes_openai.cpp
@@ -158,18 +158,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
request.gen_params.batch_count = n;
std::string sd_cpp_extra_args_str = extract_and_remove_sd_cpp_extra_args(request.gen_params.prompt);
- if (!sd_cpp_extra_args_str.empty()) {
- const json extra_args = json::parse(sd_cpp_extra_args_str, nullptr, false);
- if (extra_args.is_discarded()) {
- error_message = "invalid sd_cpp_extra_args";
- return false;
- }
- // Resolve resizing before decoding while keeping embedded image overrides last.
- if (extra_args.contains("auto_resize_ref_image") && extra_args["auto_resize_ref_image"].is_boolean()) {
- request.gen_params.auto_resize_ref_image = extra_args["auto_resize_ref_image"].get();
- }
- }
-
for (auto& bytes : images_bytes) {
int img_w = 0;
int img_h = 0;
@@ -178,12 +166,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
reinterpret_cast(bytes.data()),
static_cast(bytes.size()),
img_w, img_h, resolved_channel,
- request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
- ? request.gen_params.width
- : 0,
- request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
- ? request.gen_params.height
- : 0,
+ 0, 0,
0);
if (raw_pixels == nullptr) {
continue;
@@ -194,23 +177,10 @@ static bool build_openai_edit_request(const httplib::Request& req,
request.gen_params.set_width_and_height_if_unset(image_owner.get().width, image_owner.get().height);
if (is_first_ref_image) {
- int init_w = 0;
- int init_h = 0;
- if (request.gen_params.width_and_height_are_set()) {
- init_w = request.gen_params.width;
- init_h = request.gen_params.height;
- }
-
- int init_img_w = 0;
- int init_img_h = 0;
- int init_resolved_channel = 0;
- uint8_t* init_pixels = load_image_from_memory(
- reinterpret_cast(bytes.data()),
- static_cast(bytes.size()),
- init_img_w, init_img_h, init_resolved_channel,
- init_w, init_h, 0);
- if (init_pixels != nullptr) {
- request.gen_params.init_image.reset({(uint32_t)init_img_w, (uint32_t)init_img_h, (uint32_t)init_resolved_channel, init_pixels});
+ request.gen_params.init_image = image_owner;
+ if (request.gen_params.init_image.get().data == nullptr) {
+ error_message = "could not allocate init image";
+ return false;
}
}
@@ -218,12 +188,6 @@ static bool build_openai_edit_request(const httplib::Request& req,
}
if (!mask_bytes.empty()) {
- int expected_width = 0;
- int expected_height = 0;
- if (request.gen_params.width_and_height_are_set()) {
- expected_width = request.gen_params.width;
- expected_height = request.gen_params.height;
- }
int mask_w = 0;
int mask_h = 0;
int mask_channel = 0;
@@ -232,7 +196,7 @@ static bool build_openai_edit_request(const httplib::Request& req,
reinterpret_cast(mask_bytes.data()),
static_cast(mask_bytes.size()),
mask_w, mask_h, mask_channel,
- expected_width, expected_height, 1);
+ 0, 0, 1);
request.gen_params.mask_image.reset({(uint32_t)mask_w, (uint32_t)mask_h, 1, mask_raw});
const sd_image_t& mask_image = request.gen_params.mask_image.get();
request.gen_params.set_width_and_height_if_unset(mask_image.width, mask_image.height);
diff --git a/examples/server/routes_sdapi.cpp b/examples/server/routes_sdapi.cpp
index 1966a08f..eaf31db7 100644
--- a/examples/server/routes_sdapi.cpp
+++ b/examples/server/routes_sdapi.cpp
@@ -80,17 +80,6 @@ static enum sample_method_t get_sdapi_sample_method(std::string name) {
return it != hardcoded.end() ? it->second : SAMPLE_METHOD_COUNT;
}
-static void assign_solid_mask(SDImageOwner& mask_owner, int width, int height) {
- const size_t pixel_count = static_cast(width) * static_cast(height);
- uint8_t* raw_mask = static_cast(malloc(pixel_count));
- if (raw_mask == nullptr) {
- mask_owner.reset({0, 0, 1, nullptr});
- return;
- }
- std::memset(raw_mask, 255, pixel_count);
- mask_owner.reset({(uint32_t)width, (uint32_t)height, 1, raw_mask});
-}
-
static bool build_sdapi_img_gen_request(const json& j,
ServerRuntime& runtime,
bool img2img,
@@ -193,15 +182,25 @@ static bool build_sdapi_img_gen_request(const json& j,
}
}
- if (img2img) {
- const int expected_width = request.gen_params.width_and_height_are_set() ? request.gen_params.width : 0;
- const int expected_height = request.gen_params.width_and_height_are_set() ? request.gen_params.height : 0;
+ if (j.contains("ref_image_args")) {
+ if (!j["ref_image_args"].is_string()) {
+ error_message = "ref_image_args must be a string";
+ return false;
+ }
+ request.gen_params.ref_image_args = j["ref_image_args"].get();
+ }
+ if (j.contains("image_preprocess") && !request.gen_params.parse_image_preprocess_json(j["image_preprocess"].dump())) {
+ error_message = "invalid image_preprocess";
+ return false;
+ }
+
+ if (img2img) {
if (j.contains("init_images") && j["init_images"].is_array() && !j["init_images"].empty()) {
if (decode_base64_image(j["init_images"][0].get(),
0,
- expected_width,
- expected_height,
+ 0,
+ 0,
request.gen_params.init_image)) {
const sd_image_t& image = request.gen_params.init_image.get();
request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -211,8 +210,8 @@ static bool build_sdapi_img_gen_request(const json& j,
if (j.contains("mask") && j["mask"].is_string()) {
if (decode_base64_image(j["mask"].get(),
1,
- expected_width,
- expected_height,
+ 0,
+ 0,
request.gen_params.mask_image)) {
const sd_image_t& image = request.gen_params.mask_image.get();
request.gen_params.set_width_and_height_if_unset(image.width, image.height);
@@ -225,9 +224,7 @@ static bool build_sdapi_img_gen_request(const json& j,
}
}
} else {
- const int resolved_width = request.gen_params.get_resolved_width();
- const int resolved_height = request.gen_params.get_resolved_height();
- assign_solid_mask(request.gen_params.mask_image, resolved_width, resolved_height);
+ request.gen_params.mask_image.reset({0, 0, 1, nullptr});
}
float denoising_strength = j.value("denoising_strength", -1.f);
@@ -244,12 +241,7 @@ static bool build_sdapi_img_gen_request(const json& j,
SDImageOwner image_owner;
if (decode_base64_image(extra_image.get(),
0,
- request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
- ? request.gen_params.width
- : 0,
- request.gen_params.auto_resize_ref_image && request.gen_params.width_and_height_are_set()
- ? request.gen_params.height
- : 0,
+ 0, 0,
image_owner)) {
const sd_image_t& image = image_owner.get();
request.gen_params.set_width_and_height_if_unset(image.width, image.height);
diff --git a/examples/server/routes_sdcpp.cpp b/examples/server/routes_sdcpp.cpp
index 7253dbf4..a1eeb775 100644
--- a/examples/server/routes_sdcpp.cpp
+++ b/examples/server/routes_sdcpp.cpp
@@ -127,7 +127,8 @@ static json make_img_gen_defaults_json(const SDGenerationParams& defaults, const
{"seed", defaults.seed},
{"batch_count", defaults.batch_count},
{"qwen_image_layers", defaults.qwen_image_layers},
- {"auto_resize_ref_image", defaults.auto_resize_ref_image},
+ {"ref_image_args", defaults.ref_image_args},
+ {"image_preprocess", defaults.image_preprocess},
{"increase_ref_index", defaults.increase_ref_index},
{"control_strength", defaults.control_strength},
{"ip_adapter_strength", defaults.ip_adapter_strength},
@@ -153,6 +154,7 @@ static json make_vid_gen_defaults_json(const SDGenerationParams& defaults, const
{"strength", defaults.strength},
{"seed", defaults.seed},
{"video_frames", defaults.video_frames},
+ {"image_preprocess", defaults.image_preprocess},
{"fps", defaults.fps},
{"moe_boundary", defaults.moe_boundary},
{"vace_strength", defaults.vace_strength},
diff --git a/include/stable-diffusion.h b/include/stable-diffusion.h
index 9173240b..64a91fcb 100644
--- a/include/stable-diffusion.h
+++ b/include/stable-diffusion.h
@@ -263,6 +263,11 @@ typedef struct {
uint8_t* data;
} sd_image_t;
+typedef struct {
+ // Semicolon-separated target=...,key=value rules. NULL preserves defaults.
+ const char* rules;
+} sd_image_preprocess_params_t;
+
typedef struct {
sd_image_t* frames;
int frame_count;
@@ -410,6 +415,7 @@ typedef struct {
int qwen_image_layers;
bool circular_x;
bool circular_y;
+ sd_image_preprocess_params_t image_preprocess;
} sd_img_gen_params_t;
typedef struct {
@@ -443,6 +449,7 @@ typedef struct {
sd_hires_params_t hires;
bool circular_x;
bool circular_y;
+ sd_image_preprocess_params_t image_preprocess;
} sd_vid_gen_params_t;
typedef struct sd_ctx_t sd_ctx_t;
diff --git a/src/detailer.cpp b/src/detailer.cpp
index bbf9c688..72d01dc6 100644
--- a/src/detailer.cpp
+++ b/src/detailer.cpp
@@ -970,6 +970,7 @@ bool adetail_image(adetailer_ctx_t* context,
generation.pm_params = {};
generation.pulid_params = {};
generation.hires.enabled = false;
+ generation.image_preprocess = {};
if (params.steps > 0) {
generation.sample_params.sample_steps = params.steps;
generation.sample_params.custom_sigmas = nullptr;
diff --git a/src/pipeline/image.cpp b/src/pipeline/image.cpp
index 4c90150e..3ef5318b 100644
--- a/src/pipeline/image.cpp
+++ b/src/pipeline/image.cpp
@@ -10,6 +10,7 @@
#include "model/vae/vae.hpp"
#include "request.h"
#include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
#include "upscaler.h"
namespace sd::pipeline {
@@ -800,6 +801,12 @@ namespace sd::pipeline {
int64_t t0 = ggml_time_ms();
sd->vae_tiling_params = sd_img_gen_params->vae_tiling_params;
GenerationRequest request(sd, sd_img_gen_params);
+ sd::ImagePreprocessor preprocessing(sd_img_gen_params->image_preprocess.rules);
+ sd_img_gen_params_t processed_params = *sd_img_gen_params;
+ if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+ return false;
+ sd_img_gen_params = &processed_params;
+ request.pm_params = processed_params.pm_params;
LOG_INFO("generate_image %dx%d", request.width, request.height);
sd->rng->manual_seed(request.seed);
diff --git a/src/pipeline/video.cpp b/src/pipeline/video.cpp
index f0631284..5f227cee 100644
--- a/src/pipeline/video.cpp
+++ b/src/pipeline/video.cpp
@@ -15,6 +15,7 @@
#include "model/vae/vae.hpp"
#include "request.h"
#include "runtime/denoiser.hpp"
+#include "runtime/image_preprocess.h"
namespace sd::pipeline {
@@ -1530,6 +1531,7 @@ namespace sd::pipeline {
img_gen_params.qwen_image_layers = 0;
img_gen_params.circular_x = sd_vid_gen_params->circular_x;
img_gen_params.circular_y = sd_vid_gen_params->circular_y;
+ img_gen_params.image_preprocess = sd_vid_gen_params->image_preprocess;
sd->animatediff_num_frames = n_frames;
bool ok = generate_image(sd, &img_gen_params, frames_out, num_frames_out);
@@ -1560,6 +1562,11 @@ namespace sd::pipeline {
sd->vae_tiling_params = sd_vid_gen_params->vae_tiling_params;
sd->apply_circular_axes(sd_vid_gen_params->circular_x, sd_vid_gen_params->circular_y);
GenerationRequest request(sd, sd_vid_gen_params);
+ sd::ImagePreprocessor preprocessing(sd_vid_gen_params->image_preprocess.rules);
+ sd_vid_gen_params_t processed_params = *sd_vid_gen_params;
+ if (!preprocessing.prepare_inputs(processed_params, request.width, request.height))
+ return false;
+ sd_vid_gen_params = &processed_params;
if (fps_out != nullptr) {
*fps_out = request.fps;
}
diff --git a/src/runtime/image_preprocess.cpp b/src/runtime/image_preprocess.cpp
new file mode 100644
index 00000000..a8df7c3b
--- /dev/null
+++ b/src/runtime/image_preprocess.cpp
@@ -0,0 +1,447 @@
+#include "image_preprocess.h"
+#include "core/util.h"
+
+#include
+#include
+
+namespace sd {
+
+ static constexpr std::pair image_targets[] = {
+ {"init", ImageTarget::Init},
+ {"end", ImageTarget::End},
+ {"mask", ImageTarget::Mask},
+ {"control", ImageTarget::Control},
+ {"ref", ImageTarget::Ref},
+ {"ip-adapter", ImageTarget::IPAdapter},
+ {"id", ImageTarget::ID},
+ {"control-frame", ImageTarget::ControlFrame},
+ };
+
+ static constexpr std::pair image_resize_modes[] = {
+ {"auto", ImageResizeMode::Auto},
+ {"none", ImageResizeMode::None},
+ {"stretch", ImageResizeMode::Stretch},
+ {"crop", ImageResizeMode::Crop},
+ {"crop-resize", ImageResizeMode::CropResize},
+ {"fit-pad", ImageResizeMode::FitPad},
+ };
+
+ template
+ static bool parse_enum(const std::string& text, const std::pair (&names)[N], T& value) {
+ for (const auto& entry : names) {
+ if (text == entry.first) {
+ value = entry.second;
+ return true;
+ }
+ }
+ return false;
+ }
+
+ template
+ static const char* enum_name(T value, const std::pair (&names)[N]) {
+ for (const auto& entry : names) {
+ if (value == entry.second)
+ return entry.first;
+ }
+ return "unknown";
+ }
+
+ template
+ static bool one_of(T value, std::initializer_list choices) {
+ return std::find(choices.begin(), choices.end(), value) != choices.end();
+ }
+
+ static bool one_of(const std::string& value, std::initializer_list choices) {
+ for (const char* choice : choices) {
+ if (value == choice)
+ return true;
+ }
+ return false;
+ }
+
+ static ImageResizeMode resolve_mode(const std::map& options, ImageResizeMode default_mode) {
+ auto it = options.find("mode");
+ ImageResizeMode mode = ImageResizeMode::Auto;
+ if (it != options.end())
+ parse_enum(it->second, image_resize_modes, mode);
+ if (mode != ImageResizeMode::Auto)
+ return mode;
+ return options.count("width") && default_mode == ImageResizeMode::None ? ImageResizeMode::Stretch : default_mode;
+ }
+
+ bool ImagePreprocessor::fail(const std::string& message) const {
+ LOG_ERROR("image preprocessing: %s", message.c_str());
+ valid_ = false;
+ return false;
+ }
+
+ ImagePreprocessor::ImagePreprocessor(const char* text) {
+ if (text == nullptr || trim(text).empty())
+ return;
+ for (const auto& part : split_string(text, ';')) {
+ ImagePreprocessRule rule;
+ std::set keys;
+ if (trim(part).empty()) {
+ fail("empty rule");
+ return;
+ }
+ for (const auto& entry : split_string(part, ',')) {
+ size_t equal = entry.find('=');
+ if (equal == std::string::npos) {
+ fail("expected key=value: " + entry);
+ return;
+ }
+ std::string key = trim(entry.substr(0, equal));
+ std::string value = trim(entry.substr(equal + 1));
+ bool ok = !value.empty() && keys.insert(key).second;
+ int number = 0;
+ if (key == "target") {
+ ok &= parse_enum(value, image_targets, rule.target);
+ } else if (key == "index") {
+ ok &= parse_strict_int(value, rule.index) && rule.index >= 0;
+ } else {
+ if (key == "mode") {
+ ImageResizeMode mode;
+ ok &= parse_enum(value, image_resize_modes, mode);
+ } else if (key == "filter") {
+ ok &= one_of(value, {"auto", "nearest", "nearest-exact", "bilinear", "bicubic", "lanczos"});
+ } else if (key == "antialias") {
+ ok &= one_of(value, {"auto", "true", "false"});
+ } else if (key == "canny") {
+ ok &= one_of(value, {"true", "false"});
+ } else if (key == "anchor") {
+ ok &= one_of(value, {"center", "top", "bottom", "left", "right"});
+ } else if (key == "width" || key == "height") {
+ ok &= parse_strict_int(value, number) && number > 0;
+ } else if (key == "pad_color") {
+ ok &= value.size() == 7 || value.size() == 9;
+ ok &= !value.empty() && value[0] == '#';
+ for (size_t i = 1; i < value.size(); ++i)
+ ok &= std::isxdigit(static_cast(value[i])) != 0;
+ } else {
+ ok = false;
+ }
+ rule.options[key] = value;
+ }
+ if (!ok) {
+ fail("invalid or duplicate option: " + entry);
+ return;
+ }
+ }
+ if (!keys.count("target") || rule.options.empty() ||
+ rule.options.count("width") != rule.options.count("height") ||
+ (rule.index >= 0 && !one_of(rule.target, {ImageTarget::Ref, ImageTarget::ID, ImageTarget::ControlFrame}))) {
+ fail("invalid target, index, or incomplete dimensions: " + part);
+ return;
+ }
+ rules_.push_back(std::move(rule));
+ }
+ for (const auto& rule : rules_) {
+ const auto options = resolve_options(rule.target, std::max(0, rule.index));
+ if (options.count("antialias") && options.at("antialias") == "true" && options.count("filter") &&
+ one_of(options.at("filter"), {"nearest", "nearest-exact"})) {
+ fail("antialias requires bilinear, bicubic, or lanczos");
+ return;
+ }
+ }
+ }
+
+ std::map ImagePreprocessor::resolve_options(ImageTarget target, int index) const {
+ std::map options;
+ for (int specificity = 0; specificity < 2; ++specificity) {
+ for (const auto& rule : rules_) {
+ if (rule.target == target &&
+ rule.index == (specificity == 0 ? -1 : index)) {
+ for (const auto& entry : rule.options)
+ options[entry.first] = entry.second;
+ }
+ }
+ }
+ return options;
+ }
+
+ bool ImagePreprocessor::validate_inputs(const sd_img_gen_params_t& params) const {
+ const std::map counts = {
+ {ImageTarget::Init, params.init_image.data != nullptr},
+ {ImageTarget::Mask, params.mask_image.data != nullptr},
+ {ImageTarget::Control, params.control_image.data != nullptr},
+ {ImageTarget::IPAdapter, params.ip_adapter_image.data != nullptr},
+ {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+ {ImageTarget::ID, params.pm_params.id_images != nullptr ? params.pm_params.id_images_count : 0},
+ };
+ for (const auto& rule : rules_) {
+ auto it = counts.find(rule.target);
+ int count = it == counts.end() ? 0 : it->second;
+ if (count <= 0 || rule.index >= count) {
+ return fail(std::string("rule targets an unavailable image: ") + enum_name(rule.target, image_targets));
+ }
+ }
+ return valid_;
+ }
+
+ bool ImagePreprocessor::validate_inputs(const sd_vid_gen_params_t& params) const {
+ const std::map counts = {
+ {ImageTarget::Init, params.init_image.data != nullptr},
+ {ImageTarget::End, params.end_image.data != nullptr},
+ {ImageTarget::Ref, params.ref_images != nullptr ? params.ref_images_count : 0},
+ {ImageTarget::ControlFrame, params.control_frames != nullptr ? params.control_frames_size : 0},
+ };
+ for (const auto& rule : rules_) {
+ auto it = counts.find(rule.target);
+ int count = it == counts.end() ? 0 : it->second;
+ if (count <= 0 || rule.index >= count)
+ return fail(std::string("rule targets an unavailable video input: ") + enum_name(rule.target, image_targets));
+ }
+ return valid_;
+ }
+
+ static int anchor_offset(int remaining, const std::string& anchor, bool horizontal) {
+ if (anchor == (horizontal ? "left" : "top"))
+ return 0;
+ if (anchor == (horizontal ? "right" : "bottom"))
+ return remaining;
+ return remaining / 2;
+ }
+
+ Tensor ImagePreprocessor::apply_transform(const Tensor& image, const std::map& options, ImageTransform p, const std::string& label, ops::InterpolateMode default_filter) const {
+ auto value = [&](const char* key, const char* fallback) {
+ auto it = options.find(key);
+ return it == options.end() ? std::string(fallback) : it->second;
+ };
+ std::string filter = value("filter", "auto");
+ ops::InterpolateMode mode = default_filter;
+ if (filter == "nearest")
+ mode = ops::InterpolateMode::Nearest;
+ if (filter == "nearest-exact")
+ mode = ops::InterpolateMode::NearestExact;
+ if (filter == "bilinear")
+ mode = ops::InterpolateMode::Bilinear;
+ if (filter == "bicubic")
+ mode = ops::InterpolateMode::Bicubic;
+ if (filter == "lanczos")
+ mode = ops::InterpolateMode::Lanczos;
+ bool filtered = ops::is_2d_filter_interpolate_mode(mode);
+ bool antialias = value("antialias", "auto") == "true" ||
+ (value("antialias", "auto") == "auto" && filtered &&
+ (p.resize_width < p.crop_width || p.resize_height < p.crop_height));
+ if (antialias && !filtered) {
+ fail(label + ": antialias requires bilinear, bicubic, or lanczos");
+ return {};
+ }
+ auto cropped = ops::slice(ops::slice(image, 0, p.x, p.x + p.crop_width), 1, p.y, p.y + p.crop_height);
+ int channels = static_cast(image.shape()[2]);
+ bool resize = p.resize_width != p.crop_width || p.resize_height != p.crop_height;
+ if (resize && channels == 4 && filtered) {
+ for (int64_t i = 0, pixels = cropped.shape()[0] * cropped.shape()[1]; i < pixels; ++i) {
+ for (int c = 0; c < 3; ++c)
+ cropped[i + c * pixels] *= cropped[i + 3 * pixels];
+ }
+ }
+ auto resized = ops::interpolate(cropped, {p.resize_width, p.resize_height, channels, 1}, mode, false, antialias);
+ if (resize && channels == 4 && filtered) {
+ for (int64_t i = 0, pixels = resized.shape()[0] * resized.shape()[1]; i < pixels; ++i) {
+ float alpha = std::clamp(resized[i + 3 * pixels], 0.f, 1.f);
+ for (int c = 0; c < 3; ++c)
+ resized[i + c * pixels] = alpha > 1e-6f ? resized[i + c * pixels] / alpha : 0.f;
+ }
+ }
+ resized = ops::clamp(resized, 0.f, 1.f);
+ Tensor output({p.width, p.height, channels, 1});
+ std::string color = value("pad_color", "#000000ff");
+ if (color.size() == 7)
+ color += "ff";
+ uint8_t rgba[4];
+ for (int c = 0; c < 4; ++c)
+ rgba[c] = static_cast(std::strtoul(color.substr(1 + c * 2, 2).c_str(), nullptr, 16));
+ for (int c = 0; c < channels; ++c) {
+ float fill = rgba[channels == 1 ? 0 : c] / 255.f;
+ for (int y = 0; y < p.height; ++y) {
+ for (int x = 0; x < p.width; ++x) {
+ output.index(x, y, c, 0) = x >= p.pad_x && x < p.pad_x + p.resize_width && y >= p.pad_y && y < p.pad_y + p.resize_height
+ ? resized.index(x - p.pad_x, y - p.pad_y, c, 0)
+ : fill;
+ }
+ }
+ }
+ LOG_INFO("preprocess %s: %dx%d crop=(%d,%d,%d,%d) resize=%dx%d pad=(%d,%d) output=%dx%d filter=%s(%d) antialias=%s",
+ label.c_str(), p.source_width, p.source_height, p.x, p.y, p.crop_width, p.crop_height,
+ p.resize_width, p.resize_height, p.pad_x, p.pad_y, p.width, p.height, filter.c_str(), static_cast(mode), BOOL_STR(antialias));
+ return output;
+ }
+
+ Tensor ImagePreprocessor::apply_geometry(const Tensor& image, ImageTarget target, int index, int width, int height, ImageResizeMode default_mode, ops::InterpolateMode default_filter, ImageTransform* plan_out) const {
+ if (!valid_ || image.empty())
+ return {};
+ const std::string label = std::string(enum_name(target, image_targets)) + "[" + std::to_string(index) + "]";
+ auto options = resolve_options(target, index);
+ if (image.dim() != 4 || image.shape()[3] != 1 || image.shape()[2] < 1 || image.shape()[2] > 4) {
+ fail(label + ": expected one image with 1 to 4 channels");
+ return {};
+ }
+ ImageTransform p;
+ p.source_width = p.crop_width = static_cast(image.shape()[0]);
+ p.source_height = p.crop_height = static_cast(image.shape()[1]);
+ int target_width = width > 0 ? width : p.source_width;
+ int target_height = height > 0 ? height : p.source_height;
+ if (options.count("width")) {
+ parse_strict_int(options.at("width"), target_width);
+ parse_strict_int(options.at("height"), target_height);
+ }
+ ImageResizeMode mode = resolve_mode(options, default_mode);
+ std::string anchor = options.count("anchor") ? options.at("anchor") : "center";
+ p.width = p.resize_width = target_width;
+ p.height = p.resize_height = target_height;
+ if (mode == ImageResizeMode::None) {
+ if (options.count("width") && (target_width != p.source_width || target_height != p.source_height)) {
+ fail(label + ": mode=none conflicts with requested dimensions");
+ return {};
+ }
+ p.width = p.resize_width = p.source_width;
+ p.height = p.resize_height = p.source_height;
+ } else if (mode == ImageResizeMode::Crop || mode == ImageResizeMode::CropResize) {
+ if (mode == ImageResizeMode::Crop) {
+ p.crop_width = target_width;
+ p.crop_height = target_height;
+ } else if (int64_t(p.source_width) * target_height > int64_t(p.source_height) * target_width) {
+ p.crop_width = std::max(1, static_cast(int64_t(p.source_height) * target_width / target_height));
+ } else {
+ p.crop_height = std::max(1, static_cast(int64_t(p.source_width) * target_height / target_width));
+ }
+ if (p.crop_width > p.source_width || p.crop_height > p.source_height) {
+ fail(label + ": crop exceeds source dimensions");
+ return {};
+ }
+ p.x = anchor_offset(p.source_width - p.crop_width, anchor, true);
+ p.y = anchor_offset(p.source_height - p.crop_height, anchor, false);
+ } else if (mode == ImageResizeMode::FitPad) {
+ double scale = std::min(double(target_width) / p.source_width, double(target_height) / p.source_height);
+ p.resize_width = std::max(1, std::min(target_width, static_cast(std::round(p.source_width * scale))));
+ p.resize_height = std::max(1, std::min(target_height, static_cast(std::round(p.source_height * scale))));
+ p.pad_x = anchor_offset(target_width - p.resize_width, anchor, true);
+ p.pad_y = anchor_offset(target_height - p.resize_height, anchor, false);
+ }
+ if (p.width <= 0 || p.height <= 0) {
+ fail(label + ": invalid output dimensions");
+ return {};
+ }
+ uint64_t max_pixels = std::min(INT64_MAX, SIZE_MAX / sizeof(float)) / static_cast(image.shape()[2]);
+ if (uint64_t(p.width) * p.height > max_pixels || uint64_t(p.resize_width) * p.resize_height > max_pixels) {
+ fail(label + ": image allocation size overflows");
+ return {};
+ }
+ if (plan_out != nullptr)
+ *plan_out = p;
+ return apply_transform(image, options, p, label, default_filter);
+ }
+
+ Tensor ImagePreprocessor::preprocess_input(sd_image_t image, ImageTarget target, int index, int width, int height) {
+ if (image.data == nullptr || image.width == 0 || image.height == 0 || image.width > INT_MAX || image.height > INT_MAX || image.channel < 1 || image.channel > 4) {
+ fail(std::string(enum_name(target, image_targets)) + ": invalid input image");
+ return {};
+ }
+ auto tensor = sd_image_to_tensor(image);
+ if (target == ImageTarget::Mask && has_init_transform_) {
+ auto options = resolve_options(target, index);
+ if (image.width != init_transform_.source_width || image.height != init_transform_.source_height) {
+ fail("mask and init source dimensions must match");
+ return {};
+ }
+ bool geometry_override = options.count("width") || options.count("anchor") ||
+ (options.count("mode") && options.at("mode") != "auto");
+ if (geometry_override) {
+ ImageTransform p;
+ auto init_options = resolve_options(ImageTarget::Init, 0);
+ ImageResizeMode default_mode = resolve_mode(init_options, ImageResizeMode::CropResize);
+ auto result = apply_geometry(tensor, target, index, init_transform_.width, init_transform_.height, default_mode, ops::InterpolateMode::NearestExact, &p);
+ if (result.empty())
+ return {};
+ const auto& q = init_transform_;
+ if (p.x != q.x || p.y != q.y || p.crop_width != q.crop_width || p.crop_height != q.crop_height ||
+ p.resize_width != q.resize_width || p.resize_height != q.resize_height || p.pad_x != q.pad_x || p.pad_y != q.pad_y || p.width != q.width || p.height != q.height) {
+ fail("mask geometry conflicts with init; configure geometry on init and filter on mask");
+ return {};
+ }
+ return result;
+ }
+ return apply_transform(tensor, options, init_transform_, "mask[0]", ops::InterpolateMode::NearestExact);
+ }
+ auto result = apply_geometry(tensor, target, index, width, height, width > 0 ? ImageResizeMode::CropResize : ImageResizeMode::None,
+ target == ImageTarget::Mask ? ops::InterpolateMode::NearestExact : ops::InterpolateMode::Nearest,
+ target == ImageTarget::Init ? &init_transform_ : nullptr);
+ if (target == ImageTarget::Init)
+ has_init_transform_ = !result.empty();
+ return result;
+ }
+
+ ImagePreprocessor::~ImagePreprocessor() {
+ for (const auto& image : owned_images_)
+ std::free(image.data);
+ }
+
+ bool ImagePreprocessor::prepare_image(sd_image_t& image, ImageTarget target, int index, int width, int height) {
+ if (image.data == nullptr)
+ return true;
+ auto options = resolve_options(target, index);
+ bool canny = options.count("canny") && options.at("canny") == "true";
+ auto tensor = preprocess_input(image, target, index, width, height);
+ if (tensor.empty())
+ return false;
+ auto output = tensor_to_sd_image(tensor);
+ if (output.data == nullptr)
+ return fail("could not allocate input preprocessing buffer");
+ owned_images_.push_back(output);
+ if (canny && !preprocess_canny(output, 0.08f, 0.08f, 0.8f, 1.f, false))
+ return fail("Canny preprocessing failed");
+ image = output;
+ return true;
+ }
+
+ bool ImagePreprocessor::prepare_array(sd_image_t*& images, int count, ImageTarget target, std::vector& storage, int width, int height) {
+ if (count < 0 || (count > 0 && images == nullptr))
+ return fail(std::string("invalid image array: ") + enum_name(target, image_targets));
+ if (count == 0)
+ return true;
+ storage.assign(images, images + count);
+ for (int i = 0; i < count; ++i) {
+ if (storage[i].data == nullptr)
+ return fail(std::string("empty image in array: ") + enum_name(target, image_targets));
+ if (!prepare_image(storage[i], target, i, width, height))
+ return false;
+ }
+ images = storage.data();
+ return true;
+ }
+
+ bool ImagePreprocessor::prepare_inputs(sd_img_gen_params_t& params, int width, int height) {
+ if (prepared_)
+ return fail("inputs have already been prepared");
+ prepared_ = true;
+ if (!valid_ || !validate_inputs(params))
+ return false;
+ if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+ !prepare_image(params.mask_image, ImageTarget::Mask, 0, width, height) ||
+ !prepare_image(params.control_image, ImageTarget::Control, 0, width, height) ||
+ !prepare_image(params.ip_adapter_image, ImageTarget::IPAdapter, 0, -1, -1) ||
+ !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+ !prepare_array(params.pm_params.id_images, params.pm_params.id_images_count, ImageTarget::ID, id_images_))
+ return false;
+ params.image_preprocess = {};
+ return true;
+ }
+
+ bool ImagePreprocessor::prepare_inputs(sd_vid_gen_params_t& params, int width, int height) {
+ if (prepared_)
+ return fail("inputs have already been prepared");
+ prepared_ = true;
+ if (!valid_ || !validate_inputs(params))
+ return false;
+ if (!prepare_image(params.init_image, ImageTarget::Init, 0, width, height) ||
+ !prepare_image(params.end_image, ImageTarget::End, 0, width, height) ||
+ !prepare_array(params.ref_images, params.ref_images_count, ImageTarget::Ref, ref_images_) ||
+ !prepare_array(params.control_frames, params.control_frames_size, ImageTarget::ControlFrame, control_frames_, width, height))
+ return false;
+ params.image_preprocess = {};
+ return true;
+ }
+
+} // namespace sd
diff --git a/src/runtime/image_preprocess.h b/src/runtime/image_preprocess.h
new file mode 100644
index 00000000..b75c4e0e
--- /dev/null
+++ b/src/runtime/image_preprocess.h
@@ -0,0 +1,88 @@
+#ifndef __SD_RUNTIME_IMAGE_PREPROCESS_H__
+#define __SD_RUNTIME_IMAGE_PREPROCESS_H__
+
+#include