mirror of
https://github.com/leejet/stable-diffusion.cpp.git
synced 2026-10-03 19:37:51 -05:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 leejet
|
19bbbca1c7 | ||
|
losewayy
|
39ada0863b | ||
|
leejet
|
4c3cf7543d | ||
|
leejet
|
510bccf330 |
@@ -64,6 +64,7 @@ API and command-line option may change frequently.***
|
||||
- [HiDream-O1-Image](./docs/hidream_o1_image.md)
|
||||
- [Ideogram4](./docs/ideogram4.md)
|
||||
- [LLaDA-Image](./docs/llada_image.md)
|
||||
- [PixArt](./docs/pixart.md)
|
||||
- [Image Edit Models](./docs/edit.md)
|
||||
- [FLUX.1-Kontext-dev](./docs/kontext.md)
|
||||
- [Qwen Image Edit series](./docs/qwen_image_edit.md)
|
||||
|
||||
@@ -21,6 +21,54 @@ CPU fallback. It excludes weights and cache buffers. Within a runner lifecycle,
|
||||
the summary is printed only on the first graph or when backend capacities or the
|
||||
segment count change.
|
||||
|
||||
## Use VAE tiling to reduce encode and decode memory usage.
|
||||
|
||||
`--vae-tiling` enables spatial tiling for both VAE encoding and decoding. The
|
||||
default tile size is 256x256 **image pixels**, independent of the VAE scale factor:
|
||||
|
||||
```shell
|
||||
--vae-tiling --vae-tile-size 256x256 --vae-tile-overlap 0.5
|
||||
```
|
||||
|
||||
`--vae-tile-size` accepts one size or `WIDTHxHEIGHT`. A zero dimension uses the
|
||||
256-pixel default. Sizes are rounded down to a multiple of the VAE scale factor
|
||||
and capped at the current input dimensions. Explicit sizes below four latent
|
||||
pixels per axis (or the full axis when it is smaller) are rejected. Encoding and
|
||||
decoding use the same spatial sizes, without an additional encoding multiplier.
|
||||
Inputs that fit within a tile are processed as one tile.
|
||||
|
||||
For a 512x512 image with the default 50% overlap, both encoding and decoding use
|
||||
3x3 tiles. A 256-pixel tile corresponds to 32 latent pixels for an 8x VAE, 16 for
|
||||
a 16x VAE, and 8 for a 32x VAE. Smaller tiles reduce each graph's memory demand,
|
||||
but overlapping work can increase processing time and tiling can affect image
|
||||
quality, especially during encoding. Use larger tiles when more context is needed.
|
||||
|
||||
`--vae-relative-tile-size` overrides the absolute size on each axis with a positive
|
||||
value. Values up to and including 1 specify a fraction of the current input size;
|
||||
values greater than 1 specify a target number of tiles per axis, accounting for
|
||||
overlap. For example, `0.5x0.5` uses half the width and height in both encode and
|
||||
decode. The target overlap is clamped to 0 through 0.5 and the actual overlap is
|
||||
adjusted to fit the image. Size and overlap options require `--vae-tiling`.
|
||||
|
||||
**Migration:** `--vae-tile-size` and the C/JSON fields `tile_size_w` and
|
||||
`tile_size_h` now use image pixels instead of latent units. The C/JSON fields
|
||||
`tile_size_x/y` have been renamed to `tile_size_w/h`, and `rel_size_x/y` to
|
||||
`rel_size_w/h`. The command-line option names are unchanged. For example, an old
|
||||
decode tile size of 32 corresponds to 256 pixels for an 8x VAE or 512 pixels for a
|
||||
16x VAE. Encoding no longer enlarges explicit or relative tile sizes.
|
||||
|
||||
The main VAE decode path retries allocation failures with smaller tiles, even
|
||||
without `--vae-tiling`. Supported video VAEs first try temporal tiling; spatial
|
||||
retries use at most 256-pixel tiles initially and then halve the effective tile
|
||||
dimensions down to the minimum size. Each spatial retry must reduce the effective
|
||||
tile size. These runtime adjustments do not change the caller's parameters.
|
||||
Execution failures are not retried, and encoding has no automatic OOM retry.
|
||||
|
||||
`--temporal-tiling` remains independent of spatial tiling. MiniMax H3 always uses
|
||||
spatial tiling (256x256 pixels and 25% overlap by default) and its own temporal
|
||||
windows. With `--vae-tiling`, its overlap follows `--vae-tile-overlap`; explicit
|
||||
spatial sizes are honored.
|
||||
|
||||
## Offload weights to the CPU to save VRAM without reducing generation speed.
|
||||
|
||||
Using `--offload-to-cpu` allows you to offload weights to the CPU, saving VRAM without reducing generation speed.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# How to Use
|
||||
|
||||
You can run PixArt-α / PixArt-Σ with stable-diffusion.cpp.
|
||||
|
||||
PixArt is a DiT-based text-to-image model family conditioned by a T5-XXL text
|
||||
encoder and a 4-channel VAE: SDXL-style for PixArt-Σ and SD1.x-style for PixArt-α.
|
||||
|
||||
## Download weights
|
||||
|
||||
- Download the transformer (diffusion model)
|
||||
- PixArt-Σ XL-2 1024-MS: https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS/tree/main/transformer
|
||||
- PixArt-α XL-2 1024-MS: https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS/tree/main/transformer
|
||||
- Download the T5-XXL text encoder
|
||||
- safetensors: https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS/tree/main/text_encoder
|
||||
- Download the VAE
|
||||
- PixArt-Σ: https://huggingface.co/PixArt-alpha/PixArt-Sigma-XL-2-1024-MS/tree/main/vae
|
||||
- PixArt-α: https://huggingface.co/PixArt-alpha/PixArt-XL-2-1024-MS/tree/main/vae
|
||||
- Use the VAE matching the checkpoint's latent space. For TAE decoding or
|
||||
preview, use TAESDXL for PixArt-Σ and TAESD for PixArt-α.
|
||||
- Tokenizer: the T5 vocabulary is embedded; no extra tokenizer file is needed.
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
.\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\pixart_sigma_xl2_1024_ms.safetensors --t5xxl ..\models\text_encoders\t5xxl.safetensors --vae ..\models\vae\pixart_vae.safetensors -p "a lovely cat" --cfg-scale 4.5 -W 1024 -H 1024 --steps 20 -v
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- The VAE scaling factor defaults to `0.13025` for PixArt-Σ. PixArt-α
|
||||
checkpoints with resolution micro-condition weights use `0.18215`.
|
||||
PixArt-α 512 has the same tensor layout as PixArt-Σ, so it requires an
|
||||
explicit override: `--model-args "pixart_vae_scale_factor=0.18215"`.
|
||||
This argument can also override the scale for other compatible checkpoints.
|
||||
- PixArt-Σ checkpoints compute 2D sincos positional embeddings at runtime;
|
||||
the trained grid is 64x64 patches with an interpolation scale of 2.
|
||||
For checkpoints trained at a different resolution, the positional embedding
|
||||
parameters can be adjusted via model args:
|
||||
`--model-args "pixart_pos_embed_base_size=<trained grid>,pixart_interpolation_scale=<scale>"`
|
||||
(e.g. `pixart_pos_embed_base_size=32,pixart_interpolation_scale=1,pixart_vae_scale_factor=0.18215` for
|
||||
PixArt-α XL-2 512).
|
||||
- Checkpoints carrying resolution/aspect-ratio micro-condition weights are
|
||||
detected but those conditions are not applied yet; a warning is logged and
|
||||
generation proceeds with the timestep embedding only.
|
||||
- The transformer predicts 8 channels (noise + learned variance); only the
|
||||
noise half is used for sampling, matching the reference implementation.
|
||||
+29
-29
@@ -1341,7 +1341,7 @@ ArgOptions SDGenerationParams::get_options() {
|
||||
&embed_image_metadata},
|
||||
{"",
|
||||
"--vae-tiling",
|
||||
"process vae in tiles to reduce memory usage",
|
||||
"process vae encode and decode in spatial tiles to reduce memory usage (default: 256x256 image pixels)",
|
||||
true,
|
||||
&vae_tiling_params.enabled},
|
||||
{"",
|
||||
@@ -1605,12 +1605,12 @@ ArgOptions SDGenerationParams::get_options() {
|
||||
size_t x_pos = tile_size_str.find('x');
|
||||
try {
|
||||
if (x_pos != std::string::npos) {
|
||||
std::string tile_x_str = tile_size_str.substr(0, x_pos);
|
||||
std::string tile_y_str = tile_size_str.substr(x_pos + 1);
|
||||
vae_tiling_params.tile_size_x = std::stoi(tile_x_str);
|
||||
vae_tiling_params.tile_size_y = std::stoi(tile_y_str);
|
||||
std::string tile_w_str = tile_size_str.substr(0, x_pos);
|
||||
std::string tile_h_str = tile_size_str.substr(x_pos + 1);
|
||||
vae_tiling_params.tile_size_w = std::stoi(tile_w_str);
|
||||
vae_tiling_params.tile_size_h = std::stoi(tile_h_str);
|
||||
} else {
|
||||
vae_tiling_params.tile_size_x = vae_tiling_params.tile_size_y = std::stoi(tile_size_str);
|
||||
vae_tiling_params.tile_size_w = vae_tiling_params.tile_size_h = std::stoi(tile_size_str);
|
||||
}
|
||||
} catch (const std::invalid_argument&) {
|
||||
return -1;
|
||||
@@ -1628,12 +1628,12 @@ ArgOptions SDGenerationParams::get_options() {
|
||||
size_t x_pos = rel_size_str.find('x');
|
||||
try {
|
||||
if (x_pos != std::string::npos) {
|
||||
std::string rel_x_str = rel_size_str.substr(0, x_pos);
|
||||
std::string rel_y_str = rel_size_str.substr(x_pos + 1);
|
||||
vae_tiling_params.rel_size_x = std::stof(rel_x_str);
|
||||
vae_tiling_params.rel_size_y = std::stof(rel_y_str);
|
||||
std::string rel_w_str = rel_size_str.substr(0, x_pos);
|
||||
std::string rel_h_str = rel_size_str.substr(x_pos + 1);
|
||||
vae_tiling_params.rel_size_w = std::stof(rel_w_str);
|
||||
vae_tiling_params.rel_size_h = std::stof(rel_h_str);
|
||||
} else {
|
||||
vae_tiling_params.rel_size_x = vae_tiling_params.rel_size_y = std::stof(rel_size_str);
|
||||
vae_tiling_params.rel_size_w = vae_tiling_params.rel_size_h = std::stof(rel_size_str);
|
||||
}
|
||||
} catch (const std::invalid_argument&) {
|
||||
return -1;
|
||||
@@ -1763,11 +1763,11 @@ ArgOptions SDGenerationParams::get_options() {
|
||||
on_scm_policy_arg},
|
||||
{"",
|
||||
"--vae-tile-size",
|
||||
"tile size for vae tiling in latent units, not image pixels, format [X]x[Y] (default: 32x32)",
|
||||
"tile size for vae encode and decode in image pixels, format [W]x[H] or [S] (default: 256x256; requires --vae-tiling)",
|
||||
on_tile_size_arg},
|
||||
{"",
|
||||
"--vae-relative-tile-size",
|
||||
"relative tile size for vae tiling, format [X]x[Y], in fraction of image size if < 1, in number of tiles per dim if >=1 (overrides --vae-tile-size)",
|
||||
"relative tile size for vae encode and decode, format [W]x[H] or [S]: <=1 is a dimension fraction, >1 a target tile count (overrides --vae-tile-size; requires --vae-tiling)",
|
||||
on_relative_tile_size_arg},
|
||||
{"",
|
||||
"--prompt-file",
|
||||
@@ -2224,20 +2224,20 @@ bool SDGenerationParams::from_json_str(
|
||||
if (tiling_json.contains("temporal_tiling") && tiling_json["temporal_tiling"].is_boolean()) {
|
||||
vae_tiling_params.temporal_tiling = tiling_json["temporal_tiling"];
|
||||
}
|
||||
if (tiling_json.contains("tile_size_x") && tiling_json["tile_size_x"].is_number_integer()) {
|
||||
vae_tiling_params.tile_size_x = tiling_json["tile_size_x"];
|
||||
if (tiling_json.contains("tile_size_w") && tiling_json["tile_size_w"].is_number_integer()) {
|
||||
vae_tiling_params.tile_size_w = tiling_json["tile_size_w"];
|
||||
}
|
||||
if (tiling_json.contains("tile_size_y") && tiling_json["tile_size_y"].is_number_integer()) {
|
||||
vae_tiling_params.tile_size_y = tiling_json["tile_size_y"];
|
||||
if (tiling_json.contains("tile_size_h") && tiling_json["tile_size_h"].is_number_integer()) {
|
||||
vae_tiling_params.tile_size_h = tiling_json["tile_size_h"];
|
||||
}
|
||||
if (tiling_json.contains("target_overlap") && tiling_json["target_overlap"].is_number()) {
|
||||
vae_tiling_params.target_overlap = tiling_json["target_overlap"];
|
||||
}
|
||||
if (tiling_json.contains("rel_size_x") && tiling_json["rel_size_x"].is_number()) {
|
||||
vae_tiling_params.rel_size_x = tiling_json["rel_size_x"];
|
||||
if (tiling_json.contains("rel_size_w") && tiling_json["rel_size_w"].is_number()) {
|
||||
vae_tiling_params.rel_size_w = tiling_json["rel_size_w"];
|
||||
}
|
||||
if (tiling_json.contains("rel_size_y") && tiling_json["rel_size_y"].is_number()) {
|
||||
vae_tiling_params.rel_size_y = tiling_json["rel_size_y"];
|
||||
if (tiling_json.contains("rel_size_h") && tiling_json["rel_size_h"].is_number()) {
|
||||
vae_tiling_params.rel_size_h = tiling_json["rel_size_h"];
|
||||
}
|
||||
if (tiling_json.contains("extra_tiling_args") && tiling_json["extra_tiling_args"].is_string()) {
|
||||
extra_tiling_args = tiling_json["extra_tiling_args"].get<std::string>();
|
||||
@@ -2934,11 +2934,11 @@ std::string SDGenerationParams::to_string() const {
|
||||
<< " vae_tiling_params: { "
|
||||
<< vae_tiling_params.enabled << ", "
|
||||
<< vae_tiling_params.temporal_tiling << ", "
|
||||
<< vae_tiling_params.tile_size_x << ", "
|
||||
<< vae_tiling_params.tile_size_y << ", "
|
||||
<< vae_tiling_params.tile_size_w << ", "
|
||||
<< vae_tiling_params.tile_size_h << ", "
|
||||
<< vae_tiling_params.target_overlap << ", "
|
||||
<< vae_tiling_params.rel_size_x << ", "
|
||||
<< vae_tiling_params.rel_size_y << ", "
|
||||
<< vae_tiling_params.rel_size_w << ", "
|
||||
<< vae_tiling_params.rel_size_h << ", "
|
||||
<< "\"" << extra_tiling_args << "\" },\n"
|
||||
<< "}";
|
||||
return oss.str();
|
||||
@@ -3140,11 +3140,11 @@ std::string build_sdcpp_image_metadata_json(const SDContextParams& ctx_params,
|
||||
root["vae_tiling"] = {
|
||||
{"enabled", gen_params.vae_tiling_params.enabled},
|
||||
{"temporal_tiling", gen_params.vae_tiling_params.temporal_tiling},
|
||||
{"tile_size_x", gen_params.vae_tiling_params.tile_size_x},
|
||||
{"tile_size_y", gen_params.vae_tiling_params.tile_size_y},
|
||||
{"tile_size_w", gen_params.vae_tiling_params.tile_size_w},
|
||||
{"tile_size_h", gen_params.vae_tiling_params.tile_size_h},
|
||||
{"target_overlap", gen_params.vae_tiling_params.target_overlap},
|
||||
{"rel_size_x", gen_params.vae_tiling_params.rel_size_x},
|
||||
{"rel_size_y", gen_params.vae_tiling_params.rel_size_y},
|
||||
{"rel_size_w", gen_params.vae_tiling_params.rel_size_w},
|
||||
{"rel_size_h", gen_params.vae_tiling_params.rel_size_h},
|
||||
{"extra_tiling_args", gen_params.extra_tiling_args},
|
||||
};
|
||||
}
|
||||
|
||||
+22
-20
@@ -524,11 +524,11 @@ Shared default fields used by both `img_gen` and `vid_gen`:
|
||||
| `vae_tiling_params` | `object` |
|
||||
| `vae_tiling_params.enabled` | `boolean` |
|
||||
| `vae_tiling_params.temporal_tiling` | `boolean` |
|
||||
| `vae_tiling_params.tile_size_x` | `integer` |
|
||||
| `vae_tiling_params.tile_size_y` | `integer` |
|
||||
| `vae_tiling_params.tile_size_w` | `integer` |
|
||||
| `vae_tiling_params.tile_size_h` | `integer` |
|
||||
| `vae_tiling_params.target_overlap` | `number` |
|
||||
| `vae_tiling_params.rel_size_x` | `number` |
|
||||
| `vae_tiling_params.rel_size_y` | `number` |
|
||||
| `vae_tiling_params.rel_size_w` | `number` |
|
||||
| `vae_tiling_params.rel_size_h` | `number` |
|
||||
| `vae_tiling_params.extra_tiling_args` | `string` |
|
||||
| `cache_mode` | `string` |
|
||||
| `cache_option` | `string` |
|
||||
@@ -537,6 +537,8 @@ Shared default fields used by both `img_gen` and `vid_gen`:
|
||||
| `output_format` | `string` |
|
||||
| `output_compression` | `integer` |
|
||||
|
||||
`vae_tiling_params.tile_size_w` and `tile_size_h` are in **image pixels**, with `0` selecting the 256-pixel default. Both encode and decode use these sizes without an encoding multiplier. Positive `rel_size_w`/`rel_size_h` values override the corresponding absolute size: values up to 1 are dimension fractions, and values greater than 1 are target tile counts. Set `enabled` to use spatial tiling. Sizes are aligned down to the VAE scale factor and capped at the input dimensions; explicit sizes below the minimum supported tile size are rejected. These fields previously used latent units; see [VAE tiling](../../docs/performance.md#use-vae-tiling-to-reduce-encode-and-decode-memory-usage) for migration and OOM retry behavior.
|
||||
|
||||
`vae_tiling_params.extra_tiling_args` accepts a key=value list. Supported video VAEs accept `temporal_tile_frames` (alias `temporal_tile_size`, default `4`) and `temporal_tile_overlap` (default `1`).
|
||||
LTX and Wan preserve causal state between temporal tiles. Hunyuan Video and TAEHV use overlap blending. MiniMax H3 keeps its model-specific fixed temporal windows because its latent-to-frame mapping is non-linear.
|
||||
|
||||
@@ -767,11 +769,11 @@ Example:
|
||||
"vae_tiling_params": {
|
||||
"enabled": false,
|
||||
"temporal_tiling": false,
|
||||
"tile_size_x": 0,
|
||||
"tile_size_y": 0,
|
||||
"tile_size_w": 0,
|
||||
"tile_size_h": 0,
|
||||
"target_overlap": 0.5,
|
||||
"rel_size_x": 0.0,
|
||||
"rel_size_y": 0.0,
|
||||
"rel_size_w": 0.0,
|
||||
"rel_size_h": 0.0,
|
||||
"extra_tiling_args": ""
|
||||
},
|
||||
|
||||
@@ -900,11 +902,11 @@ Other native fields:
|
||||
| `vae_tiling_params` | `object` |
|
||||
| `vae_tiling_params.enabled` | `boolean` |
|
||||
| `vae_tiling_params.temporal_tiling` | `boolean` |
|
||||
| `vae_tiling_params.tile_size_x` | `integer` |
|
||||
| `vae_tiling_params.tile_size_y` | `integer` |
|
||||
| `vae_tiling_params.tile_size_w` | `integer` |
|
||||
| `vae_tiling_params.tile_size_h` | `integer` |
|
||||
| `vae_tiling_params.target_overlap` | `number` |
|
||||
| `vae_tiling_params.rel_size_x` | `number` |
|
||||
| `vae_tiling_params.rel_size_y` | `number` |
|
||||
| `vae_tiling_params.rel_size_w` | `number` |
|
||||
| `vae_tiling_params.rel_size_h` | `number` |
|
||||
| `vae_tiling_params.extra_tiling_args` | `string` |
|
||||
| `cache_mode` | `string` |
|
||||
| `cache_option` | `string` |
|
||||
@@ -1115,11 +1117,11 @@ Example:
|
||||
"vae_tiling_params": {
|
||||
"enabled": false,
|
||||
"temporal_tiling": false,
|
||||
"tile_size_x": 0,
|
||||
"tile_size_y": 0,
|
||||
"tile_size_w": 0,
|
||||
"tile_size_h": 0,
|
||||
"target_overlap": 0.5,
|
||||
"rel_size_x": 0.0,
|
||||
"rel_size_y": 0.0,
|
||||
"rel_size_w": 0.0,
|
||||
"rel_size_h": 0.0,
|
||||
"extra_tiling_args": ""
|
||||
},
|
||||
|
||||
@@ -1240,11 +1242,11 @@ Other native fields:
|
||||
| `vae_tiling_params` | `object` |
|
||||
| `vae_tiling_params.enabled` | `boolean` |
|
||||
| `vae_tiling_params.temporal_tiling` | `boolean` |
|
||||
| `vae_tiling_params.tile_size_x` | `integer` |
|
||||
| `vae_tiling_params.tile_size_y` | `integer` |
|
||||
| `vae_tiling_params.tile_size_w` | `integer` |
|
||||
| `vae_tiling_params.tile_size_h` | `integer` |
|
||||
| `vae_tiling_params.target_overlap` | `number` |
|
||||
| `vae_tiling_params.rel_size_x` | `number` |
|
||||
| `vae_tiling_params.rel_size_y` | `number` |
|
||||
| `vae_tiling_params.rel_size_w` | `number` |
|
||||
| `vae_tiling_params.rel_size_h` | `number` |
|
||||
| `vae_tiling_params.extra_tiling_args` | `string` |
|
||||
| `cache_mode` | `string` |
|
||||
| `cache_option` | `string` |
|
||||
|
||||
+1
-1
Submodule examples/server/frontend updated: c4bce3d6b3...dd74a8e808
@@ -78,11 +78,11 @@ static json make_vae_tiling_json(const sd_tiling_params_t& params) {
|
||||
return {
|
||||
{"enabled", params.enabled},
|
||||
{"temporal_tiling", params.temporal_tiling},
|
||||
{"tile_size_x", params.tile_size_x},
|
||||
{"tile_size_y", params.tile_size_y},
|
||||
{"tile_size_w", params.tile_size_w},
|
||||
{"tile_size_h", params.tile_size_h},
|
||||
{"target_overlap", params.target_overlap},
|
||||
{"rel_size_x", params.rel_size_x},
|
||||
{"rel_size_y", params.rel_size_y},
|
||||
{"rel_size_w", params.rel_size_w},
|
||||
{"rel_size_h", params.rel_size_h},
|
||||
{"extra_tiling_args", params.extra_tiling_args ? params.extra_tiling_args : ""},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -173,11 +173,13 @@ enum lora_apply_mode_t {
|
||||
typedef struct {
|
||||
bool enabled;
|
||||
bool temporal_tiling;
|
||||
int tile_size_x;
|
||||
int tile_size_y;
|
||||
// Spatial tile dimensions in image pixels for both encode and decode; 0 uses 256.
|
||||
int tile_size_w;
|
||||
int tile_size_h;
|
||||
float target_overlap;
|
||||
float rel_size_x;
|
||||
float rel_size_y;
|
||||
// Positive values override tile_size: <= 1 is a dimension fraction, > 1 a target tile count.
|
||||
float rel_size_w;
|
||||
float rel_size_h;
|
||||
const char* extra_tiling_args;
|
||||
} sd_tiling_params_t;
|
||||
|
||||
|
||||
+30
-13
@@ -478,7 +478,12 @@ namespace sd::backend_fit {
|
||||
return true;
|
||||
}
|
||||
|
||||
bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params, bool prefer_temporal_tiling, ggml_status status) {
|
||||
bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
|
||||
bool prefer_temporal_tiling,
|
||||
ggml_status status,
|
||||
int latent_tile_size_w,
|
||||
int latent_tile_size_h,
|
||||
int scale_factor) {
|
||||
// Execution failures can leave the device unusable; tiling only helps with allocation failures.
|
||||
if (status != GGML_STATUS_ALLOC_FAILED) {
|
||||
return false;
|
||||
@@ -487,19 +492,31 @@ namespace sd::backend_fit {
|
||||
if (prefer_temporal_tiling && !tiling_params.temporal_tiling) {
|
||||
tiling_params.temporal_tiling = true;
|
||||
retry_mode = tiling_params.enabled ? "spatial+temporal" : "temporal";
|
||||
} else if (!tiling_params.enabled) {
|
||||
tiling_params.enabled = true;
|
||||
tiling_params.rel_size_x = 0.5f;
|
||||
tiling_params.rel_size_y = 0.5f;
|
||||
if (tiling_params.tile_size_x <= 0) {
|
||||
tiling_params.tile_size_x = 256;
|
||||
}
|
||||
if (tiling_params.tile_size_y <= 0) {
|
||||
tiling_params.tile_size_y = 256;
|
||||
}
|
||||
retry_mode = tiling_params.temporal_tiling ? "spatial+temporal" : "spatial";
|
||||
} else {
|
||||
return false;
|
||||
if (latent_tile_size_w <= 0 || latent_tile_size_h <= 0 || scale_factor <= 0) {
|
||||
return false;
|
||||
}
|
||||
auto smaller_tile = [&](int size) {
|
||||
int next_size = size / 2;
|
||||
if (!tiling_params.enabled) {
|
||||
next_size = std::min(next_size, 256 / scale_factor);
|
||||
}
|
||||
return std::min(size, std::max(4, next_size));
|
||||
};
|
||||
const int tile_size_w = smaller_tile(latent_tile_size_w);
|
||||
const int tile_size_h = smaller_tile(latent_tile_size_h);
|
||||
if (tile_size_w == latent_tile_size_w && tile_size_h == latent_tile_size_h) {
|
||||
return false;
|
||||
}
|
||||
tiling_params.enabled = true;
|
||||
tiling_params.rel_size_w = 0.0f;
|
||||
tiling_params.rel_size_h = 0.0f;
|
||||
tiling_params.tile_size_w = tile_size_w * scale_factor;
|
||||
tiling_params.tile_size_h = tile_size_h * scale_factor;
|
||||
retry_mode = tiling_params.temporal_tiling ? "spatial+temporal" : "spatial";
|
||||
LOG_WARN("Reducing VAE decode tiles from %dx%d to %dx%d image pixels",
|
||||
latent_tile_size_w * scale_factor, latent_tile_size_h * scale_factor,
|
||||
tiling_params.tile_size_w, tiling_params.tile_size_h);
|
||||
}
|
||||
|
||||
LOG_WARN("VAE decode ran out of memory; retrying with %s tiling",
|
||||
|
||||
@@ -17,7 +17,10 @@ namespace sd::backend_fit {
|
||||
|
||||
bool prepare_vae_decode_retry_tiling(sd_tiling_params_t& tiling_params,
|
||||
bool prefer_temporal_tiling,
|
||||
ggml_status status);
|
||||
ggml_status status,
|
||||
int latent_tile_size_w,
|
||||
int latent_tile_size_h,
|
||||
int scale_factor);
|
||||
|
||||
} // namespace sd::backend_fit
|
||||
|
||||
|
||||
+7
-1
@@ -62,6 +62,7 @@ enum SDVersion {
|
||||
VERSION_SENSENOVA_U1_5,
|
||||
VERSION_LLADA_IMAGE,
|
||||
VERSION_ESRGAN,
|
||||
VERSION_PIXART,
|
||||
VERSION_COUNT,
|
||||
};
|
||||
|
||||
@@ -252,6 +253,10 @@ static inline bool sd_version_is_sensenova_u1(SDVersion version) {
|
||||
return version == VERSION_SENSENOVA_U1_5;
|
||||
}
|
||||
|
||||
static inline bool sd_version_is_pixart(SDVersion version) {
|
||||
return version == VERSION_PIXART;
|
||||
}
|
||||
|
||||
static inline bool sd_version_supports_video_generation(SDVersion version) {
|
||||
return version == VERSION_SVD || sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_lingbot_video(version) || sd_version_is_ltxav(version) || sd_version_is_minimax_h3(version);
|
||||
}
|
||||
@@ -320,7 +325,8 @@ static inline bool sd_version_is_dit(SDVersion version) {
|
||||
sd_version_is_sefi_image(version) ||
|
||||
sd_version_is_krea2(version) ||
|
||||
sd_version_is_mage_flow(version) ||
|
||||
sd_version_is_sensenova_u1(version)) {
|
||||
sd_version_is_sensenova_u1(version) ||
|
||||
sd_version_is_pixart(version)) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
|
||||
@@ -163,6 +163,27 @@ struct LoraModel : public GGMLRunner {
|
||||
|
||||
lora_tensors = std::move(new_lora_tensors);
|
||||
}
|
||||
|
||||
std::unordered_map<std::string, ggml_tensor*> new_lora_tensors;
|
||||
for (const auto& [old_name, tensor] : lora_tensors) {
|
||||
std::string new_name = old_name;
|
||||
if (starts_with(old_name, "lora.model.diffusion_model.transformer_blocks.")) {
|
||||
// Qwen Image 2.1 stores the gate before the projection in fused MLP weights.
|
||||
for (const auto& suffix : {std::string(".img_mlp.gate_layer.weight."), std::string(".img_mlp.proj.weight.")}) {
|
||||
size_t pos = old_name.find(suffix);
|
||||
if (pos == std::string::npos) {
|
||||
continue;
|
||||
}
|
||||
std::string fused_name = old_name.substr(5, pos - 5) + ".img_mlp.gate_up.weight";
|
||||
if (model_tensor_names.find(fused_name) != model_tensor_names.end()) {
|
||||
new_name = "lora." + fused_name + (suffix == ".img_mlp.proj.weight." ? ".1." : ".") + old_name.substr(pos + suffix.size());
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
new_lora_tensors[new_name] = tensor;
|
||||
}
|
||||
lora_tensors = std::move(new_lora_tensors);
|
||||
}
|
||||
|
||||
ggml_tensor* get_lora_weight_diff(const std::string& model_tensor_name, ggml_context* ctx, ggml_backend_t backend) {
|
||||
|
||||
@@ -0,0 +1,390 @@
|
||||
#ifndef __SD_MODEL_DIFFUSION_PIXART_HPP__
|
||||
#define __SD_MODEL_DIFFUSION_PIXART_HPP__
|
||||
|
||||
#include <algorithm>
|
||||
#include <cinttypes>
|
||||
#include <cmath>
|
||||
#include <vector>
|
||||
|
||||
#include "core/ggml_extend.h"
|
||||
#include "core/ggml_runner.h"
|
||||
#include "core/util.h"
|
||||
#include "model/common/ggml_block.hpp"
|
||||
#include "model/diffusion/dit.hpp"
|
||||
#include "model/diffusion/mmdit.hpp"
|
||||
#include "model/diffusion/model.hpp"
|
||||
#include "model_loader.h"
|
||||
|
||||
// Ref: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/pixart_transformer_2d.py
|
||||
// Ref: https://github.com/PixArt-alpha/PixArt-sigma
|
||||
|
||||
namespace PixArt {
|
||||
constexpr int PIXART_GRAPH_SIZE = 20480;
|
||||
constexpr int ADALN_EMBED_DIM = 256;
|
||||
|
||||
struct PixArtConfig {
|
||||
int64_t in_channels = 4;
|
||||
int64_t out_channels = 8; // learn_sigma: noise prediction + learned variance
|
||||
int64_t hidden_size = 1152;
|
||||
int64_t cross_attention_dim = 1152;
|
||||
int64_t caption_channels = 4096;
|
||||
int64_t num_heads = 16;
|
||||
int64_t patch_size = 2;
|
||||
int64_t ffn_dim = 4608;
|
||||
int64_t pos_embed_base_size = 64;
|
||||
float interpolation_scale = 2.f;
|
||||
int num_layers = 28;
|
||||
|
||||
static PixArtConfig detect_from_weights(const String2TensorStorage& weights, const std::string& prefix) {
|
||||
PixArtConfig config;
|
||||
auto find = [&](const std::string& suffix) -> const TensorStorage* {
|
||||
auto it = weights.find(prefix + "." + suffix);
|
||||
return it == weights.end() ? nullptr : &it->second;
|
||||
};
|
||||
if (auto w = find("pos_embed.proj.weight")) {
|
||||
config.hidden_size = w->ne[3];
|
||||
config.in_channels = w->ne[2];
|
||||
config.patch_size = w->ne[0];
|
||||
}
|
||||
if (auto w = find("proj_out.weight")) {
|
||||
config.out_channels = w->ne[1] / (config.patch_size * config.patch_size);
|
||||
}
|
||||
if (auto w = find("caption_projection.linear_1.weight")) {
|
||||
config.caption_channels = w->ne[0];
|
||||
}
|
||||
if (auto w = find("transformer_blocks.0.attn2.to_k.weight")) {
|
||||
config.cross_attention_dim = w->ne[0];
|
||||
}
|
||||
if (auto w = find("transformer_blocks.0.ff.net.0.proj.weight")) {
|
||||
config.ffn_dim = w->ne[1];
|
||||
}
|
||||
if (find("adaln_single.emb.resolution_embedder.linear_1.weight") != nullptr) {
|
||||
LOG_WARN("pixart: resolution/aspect-ratio micro conditions are not supported; output may differ from the reference");
|
||||
}
|
||||
int layers = 0;
|
||||
const std::string block_prefix = prefix + ".transformer_blocks.";
|
||||
for (const auto& [name, _] : weights) {
|
||||
if (starts_with(name, block_prefix)) {
|
||||
layers = std::max(layers, atoi(name.substr(block_prefix.size()).c_str()) + 1);
|
||||
}
|
||||
}
|
||||
if (layers > 0) {
|
||||
config.num_layers = layers;
|
||||
LOG_VERBOSE("pixart: layers = %d, hidden_size = %" PRId64,
|
||||
layers, config.hidden_size);
|
||||
}
|
||||
return config;
|
||||
}
|
||||
};
|
||||
|
||||
// Mirrors diffusers get_2d_sincos_pos_embed for a (gh, gw) patch grid.
|
||||
static std::vector<float> gen_2d_sincos_pos_embed(int64_t dim,
|
||||
int64_t gh,
|
||||
int64_t gw,
|
||||
int64_t base_size,
|
||||
float interpolation_scale) {
|
||||
// diffusers: meshgrid(grid_w, grid_h, indexing="xy") -> grid[0]=w, grid[1]=h,
|
||||
// embedding = concat(sincos(w), sincos(h))
|
||||
std::vector<float> out(static_cast<size_t>(gh) * gw * dim);
|
||||
int64_t quarter = dim / 4;
|
||||
for (int64_t h = 0; h < gh; ++h) {
|
||||
float pos_h = static_cast<float>(h) / (static_cast<float>(gh) / base_size) / interpolation_scale;
|
||||
for (int64_t w = 0; w < gw; ++w) {
|
||||
float pos_w = static_cast<float>(w) / (static_cast<float>(gw) / base_size) / interpolation_scale;
|
||||
float* dst_w = out.data() + (h * gw + w) * dim;
|
||||
float* dst_h = dst_w + dim / 2;
|
||||
for (int64_t i = 0; i < quarter; ++i) {
|
||||
float omega = 1.f / powf(10000.f, static_cast<float>(i) / quarter);
|
||||
dst_w[i] = sinf(pos_w * omega);
|
||||
dst_w[i + quarter] = cosf(pos_w * omega);
|
||||
dst_h[i] = sinf(pos_h * omega);
|
||||
dst_h[i + quarter] = cosf(pos_h * omega);
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
class PixArtTimestepEmbedding : public GGMLBlock {
|
||||
public:
|
||||
PixArtTimestepEmbedding(int64_t in_channels, int64_t out_dim) {
|
||||
blocks["linear_1"] = std::make_shared<Linear>(in_channels, out_dim);
|
||||
blocks["linear_2"] = std::make_shared<Linear>(out_dim, out_dim);
|
||||
}
|
||||
|
||||
ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) {
|
||||
x = std::dynamic_pointer_cast<Linear>(blocks["linear_1"])->forward(ctx, x);
|
||||
x = ggml_silu(ctx->ggml_ctx, x);
|
||||
return std::dynamic_pointer_cast<Linear>(blocks["linear_2"])->forward(ctx, x);
|
||||
}
|
||||
};
|
||||
|
||||
class PixArtAttention : public GGMLBlock {
|
||||
int64_t num_heads;
|
||||
|
||||
public:
|
||||
PixArtAttention(int64_t dim, int64_t num_heads, int64_t context_dim)
|
||||
: num_heads(num_heads) {
|
||||
blocks["to_q"] = std::make_shared<Linear>(dim, dim);
|
||||
blocks["to_k"] = std::make_shared<Linear>(context_dim, dim);
|
||||
blocks["to_v"] = std::make_shared<Linear>(context_dim, dim);
|
||||
blocks["to_out.0"] = std::make_shared<Linear>(dim, dim);
|
||||
}
|
||||
|
||||
ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* context, ggml_tensor* mask = nullptr) {
|
||||
// x: [N, n_token, dim], context: [N, n_context, context_dim]
|
||||
auto q = std::dynamic_pointer_cast<Linear>(blocks["to_q"])->forward(ctx, x);
|
||||
auto k = std::dynamic_pointer_cast<Linear>(blocks["to_k"])->forward(ctx, context);
|
||||
auto v = std::dynamic_pointer_cast<Linear>(blocks["to_v"])->forward(ctx, context);
|
||||
|
||||
auto out = ggml_ext_attention_ext(ctx, q, k, v, num_heads, mask, false, ctx->flash_attn_enabled);
|
||||
return std::dynamic_pointer_cast<Linear>(blocks["to_out.0"])->forward(ctx, out);
|
||||
}
|
||||
};
|
||||
|
||||
class PixArtBlock : public GGMLBlock {
|
||||
int64_t dim;
|
||||
|
||||
void init_params(ggml_context* ctx,
|
||||
const String2TensorStorage& tensor_storage_map = {},
|
||||
const std::string prefix = "") override {
|
||||
ggml_type wtype = get_type(prefix + "scale_shift_table", tensor_storage_map, GGML_TYPE_F32);
|
||||
params["scale_shift_table"] = ggml_new_tensor_2d(ctx, wtype, dim, 6);
|
||||
}
|
||||
|
||||
public:
|
||||
PixArtBlock(int64_t dim, int64_t num_heads, int64_t context_dim, int64_t ffn_dim)
|
||||
: dim(dim) {
|
||||
blocks["attn1"] = std::make_shared<PixArtAttention>(dim, num_heads, dim);
|
||||
blocks["attn2"] = std::make_shared<PixArtAttention>(dim, num_heads, context_dim);
|
||||
blocks["ff.net.0.proj"] = std::make_shared<Linear>(dim, ffn_dim);
|
||||
blocks["ff.net.2"] = std::make_shared<Linear>(ffn_dim, dim);
|
||||
}
|
||||
|
||||
static ggml_tensor* norm(ggml_context* ctx, ggml_tensor* x) {
|
||||
return ggml_norm(ctx, x, 1e-6f);
|
||||
}
|
||||
|
||||
ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x, ggml_tensor* mod, ggml_tensor* context, ggml_tensor* context_mask) {
|
||||
// x: [N, n_token, dim]
|
||||
// mod: [N, 6 * dim], shared adaLN-single output
|
||||
int64_t N = x->ne[2];
|
||||
|
||||
auto table = params["scale_shift_table"];
|
||||
if (table->type != GGML_TYPE_F32) {
|
||||
table = ggml_cast(ctx->ggml_ctx, table, GGML_TYPE_F32);
|
||||
}
|
||||
table = ggml_reshape_3d(ctx->ggml_ctx, table, dim, 6, 1);
|
||||
auto m = ggml_add(ctx->ggml_ctx, ggml_reshape_3d(ctx->ggml_ctx, mod, dim, 6, N), table);
|
||||
auto mv = ggml_ext_chunk(ctx->ggml_ctx, ggml_reshape_2d(ctx->ggml_ctx, ggml_ext_cont(ctx->ggml_ctx, m), dim * 6, N), 6, 0);
|
||||
|
||||
auto attn1 = std::dynamic_pointer_cast<PixArtAttention>(blocks["attn1"]);
|
||||
auto attn2 = std::dynamic_pointer_cast<PixArtAttention>(blocks["attn2"]);
|
||||
auto proj = std::dynamic_pointer_cast<Linear>(blocks["ff.net.0.proj"]);
|
||||
auto fc2 = std::dynamic_pointer_cast<Linear>(blocks["ff.net.2"]);
|
||||
|
||||
auto gate = [&](ggml_tensor* y, ggml_tensor* g) {
|
||||
g = ggml_reshape_3d(ctx->ggml_ctx, g, dim, 1, N);
|
||||
return ggml_add(ctx->ggml_ctx, x, ggml_mul(ctx->ggml_ctx, y, g));
|
||||
};
|
||||
|
||||
auto h = modulate(ctx->ggml_ctx, norm(ctx->ggml_ctx, x), mv[0], mv[1]);
|
||||
x = gate(attn1->forward(ctx, h, h), mv[2]);
|
||||
// ada_norm_single: no norm before cross-attention (PixArtMS.py)
|
||||
x = ggml_add(ctx->ggml_ctx, x, attn2->forward(ctx, x, context, context_mask));
|
||||
h = modulate(ctx->ggml_ctx, norm(ctx->ggml_ctx, x), mv[3], mv[4]);
|
||||
h = proj->forward(ctx, h);
|
||||
h = ggml_ext_gelu(ctx->ggml_ctx, h, true);
|
||||
h = fc2->forward(ctx, h);
|
||||
return gate(h, mv[5]);
|
||||
}
|
||||
};
|
||||
|
||||
class PixArtModel : public GGMLBlock {
|
||||
PixArtConfig config;
|
||||
|
||||
void init_params(ggml_context* ctx,
|
||||
const String2TensorStorage& tensor_storage_map = {},
|
||||
const std::string prefix = "") override {
|
||||
ggml_type wtype = get_type(prefix + "scale_shift_table", tensor_storage_map, GGML_TYPE_F32);
|
||||
params["scale_shift_table"] = ggml_new_tensor_2d(ctx, wtype, config.hidden_size, 2);
|
||||
}
|
||||
|
||||
public:
|
||||
PixArtModel() = default;
|
||||
PixArtModel(const PixArtConfig& config)
|
||||
: config(config) {
|
||||
blocks["pos_embed.proj"] = std::make_shared<Conv2d>(config.in_channels,
|
||||
config.hidden_size,
|
||||
std::pair<int, int>{static_cast<int>(config.patch_size), static_cast<int>(config.patch_size)},
|
||||
std::pair<int, int>{static_cast<int>(config.patch_size), static_cast<int>(config.patch_size)});
|
||||
blocks["adaln_single.emb.timestep_embedder"] = std::make_shared<PixArtTimestepEmbedding>(ADALN_EMBED_DIM, config.hidden_size);
|
||||
blocks["adaln_single.linear"] = std::make_shared<Linear>(config.hidden_size, 6 * config.hidden_size);
|
||||
blocks["caption_projection.linear_1"] = std::make_shared<Linear>(config.caption_channels, config.hidden_size);
|
||||
blocks["caption_projection.linear_2"] = std::make_shared<Linear>(config.hidden_size, config.cross_attention_dim);
|
||||
for (int i = 0; i < config.num_layers; ++i) {
|
||||
blocks["transformer_blocks." + std::to_string(i)] =
|
||||
std::make_shared<PixArtBlock>(config.hidden_size, config.num_heads, config.cross_attention_dim, config.ffn_dim);
|
||||
}
|
||||
blocks["norm_out"] = std::make_shared<LayerNorm>(config.hidden_size, 1e-6f, false);
|
||||
blocks["proj_out"] = std::make_shared<Linear>(config.hidden_size,
|
||||
config.patch_size * config.patch_size * config.out_channels);
|
||||
}
|
||||
|
||||
ggml_tensor* forward(GGMLRunnerContext* ctx,
|
||||
ggml_tensor* x,
|
||||
ggml_tensor* timesteps,
|
||||
ggml_tensor* context,
|
||||
ggml_tensor* pos_embed,
|
||||
ggml_tensor* context_mask) {
|
||||
// x: [N, C, H, W] latent, context: [N, n_ctx, caption_channels]
|
||||
int64_t W = x->ne[0];
|
||||
int64_t H = x->ne[1];
|
||||
int64_t N = x->ne[3];
|
||||
int64_t p = config.patch_size;
|
||||
int64_t wp = W / p;
|
||||
int64_t hp = H / p;
|
||||
|
||||
auto h = std::dynamic_pointer_cast<Conv2d>(blocks["pos_embed.proj"])->forward(ctx, x); // [N, hidden, hp, wp]
|
||||
h = ggml_ext_cont(ctx->ggml_ctx, ggml_permute(ctx->ggml_ctx, h, 1, 2, 0, 3)); // [N, hp, wp, hidden] -> [N, hp*wp, hidden]
|
||||
h = ggml_reshape_3d(ctx->ggml_ctx, h, config.hidden_size, wp * hp, N); // [N, hp*wp, hidden]
|
||||
h = ggml_add(ctx->ggml_ctx, h, pos_embed);
|
||||
|
||||
auto t = ggml_ext_timestep_embedding(ctx->ggml_ctx, timesteps, ADALN_EMBED_DIM, 10000);
|
||||
auto emb = std::dynamic_pointer_cast<PixArtTimestepEmbedding>(blocks["adaln_single.emb.timestep_embedder"])->forward(ctx, t);
|
||||
|
||||
auto mod = std::dynamic_pointer_cast<Linear>(blocks["adaln_single.linear"])
|
||||
->forward(ctx, ggml_silu(ctx->ggml_ctx, emb)); // [N, 6 * hidden]
|
||||
|
||||
auto ctx_emb = std::dynamic_pointer_cast<Linear>(blocks["caption_projection.linear_1"])->forward(ctx, context);
|
||||
ctx_emb = ggml_ext_gelu(ctx->ggml_ctx, ctx_emb, true);
|
||||
ctx_emb = std::dynamic_pointer_cast<Linear>(blocks["caption_projection.linear_2"])->forward(ctx, ctx_emb);
|
||||
|
||||
for (int i = 0; i < config.num_layers; ++i) {
|
||||
auto block = std::dynamic_pointer_cast<PixArtBlock>(blocks["transformer_blocks." + std::to_string(i)]);
|
||||
h = block->forward(ctx, h, mod, ctx_emb, context_mask);
|
||||
sd::ggml_graph_cut::mark_graph_cut(h, "pixart.transformer_blocks." + std::to_string(i), "h");
|
||||
}
|
||||
|
||||
// scale_shift_table + emb -> (shift, scale) for the affine-free final norm
|
||||
auto tail_table = params["scale_shift_table"];
|
||||
if (tail_table->type != GGML_TYPE_F32) {
|
||||
tail_table = ggml_cast(ctx->ggml_ctx, tail_table, GGML_TYPE_F32);
|
||||
}
|
||||
auto ss = ggml_add(ctx->ggml_ctx,
|
||||
ggml_reshape_3d(ctx->ggml_ctx, tail_table, config.hidden_size, 2, 1),
|
||||
ggml_reshape_3d(ctx->ggml_ctx, emb, config.hidden_size, 1, N)); // [2, hidden, N]
|
||||
auto parts = ggml_ext_chunk(ctx->ggml_ctx,
|
||||
ggml_reshape_2d(ctx->ggml_ctx, ggml_ext_cont(ctx->ggml_ctx, ss), config.hidden_size * 2, N),
|
||||
2, 0);
|
||||
h = std::dynamic_pointer_cast<LayerNorm>(blocks["norm_out"])->forward(ctx, h);
|
||||
h = modulate(ctx->ggml_ctx, h, parts[0], parts[1]);
|
||||
h = std::dynamic_pointer_cast<Linear>(blocks["proj_out"])->forward(ctx, h); // [N, hp*wp, p*p*out_ch]
|
||||
h = DiT::unpatchify(ctx->ggml_ctx, h, hp, wp, static_cast<int>(p), static_cast<int>(p), false);
|
||||
return h; // [N, out_channels, H, W]
|
||||
}
|
||||
};
|
||||
|
||||
struct PixArtRunner : public DiffusionModelRunner {
|
||||
PixArtConfig config;
|
||||
PixArtModel model;
|
||||
std::vector<float> pos_vec;
|
||||
|
||||
PixArtRunner(ggml_backend_t backend,
|
||||
const String2TensorStorage& tensor_storage_map = {},
|
||||
const std::string prefix = "",
|
||||
std::shared_ptr<RunnerWeightManager> weight_manager = nullptr,
|
||||
const char* model_args = nullptr)
|
||||
: DiffusionModelRunner(backend, prefix, weight_manager),
|
||||
config(PixArtConfig::detect_from_weights(tensor_storage_map, prefix)) {
|
||||
for (const auto& [key, value] : parse_key_value_args(model_args, "model arg")) {
|
||||
if (key == "pixart_pos_embed_base_size") {
|
||||
int parsed = 0;
|
||||
if (parse_strict_int(value, parsed)) {
|
||||
config.pos_embed_base_size = parsed;
|
||||
} else {
|
||||
LOG_WARN("ignoring invalid PixArt model arg '%s=%s'", key.c_str(), value.c_str());
|
||||
}
|
||||
} else if (key == "pixart_interpolation_scale") {
|
||||
float parsed = 0.f;
|
||||
if (parse_strict_float(value, parsed)) {
|
||||
config.interpolation_scale = parsed;
|
||||
} else {
|
||||
LOG_WARN("ignoring invalid PixArt model arg '%s=%s'", key.c_str(), value.c_str());
|
||||
}
|
||||
}
|
||||
}
|
||||
model = PixArtModel(config);
|
||||
model.init(params_ctx, tensor_storage_map, prefix);
|
||||
}
|
||||
|
||||
std::string get_desc() override {
|
||||
return "pixart";
|
||||
}
|
||||
|
||||
void get_param_tensors(std::map<std::string, ggml_tensor*>& tensors, const std::string& prefix) override {
|
||||
model.get_param_tensors(tensors, prefix);
|
||||
}
|
||||
|
||||
ggml_cgraph* build_graph(const sd::Tensor<float>& x_tensor,
|
||||
const sd::Tensor<float>& timesteps_tensor,
|
||||
const sd::Tensor<float>& context_tensor,
|
||||
const sd::Tensor<float>& mask_tensor) {
|
||||
ggml_cgraph* gf = new_graph_custom(PIXART_GRAPH_SIZE);
|
||||
ggml_tensor* x = make_input(x_tensor);
|
||||
ggml_tensor* timesteps = make_input(timesteps_tensor);
|
||||
GGML_ASSERT(x->ne[3] == 1);
|
||||
GGML_ASSERT(!context_tensor.empty());
|
||||
ggml_tensor* context = make_input(context_tensor);
|
||||
|
||||
ggml_tensor* context_mask = nullptr;
|
||||
if (!mask_tensor.empty()) {
|
||||
// additive attention bias over context tokens: 0 keep / -inf discard
|
||||
context_mask = ggml_reshape_4d(compute_ctx, make_input(mask_tensor), mask_tensor.shape()[0], 1, 1, 1);
|
||||
}
|
||||
|
||||
int64_t W = x->ne[0];
|
||||
int64_t H = x->ne[1];
|
||||
int64_t wp = W / config.patch_size;
|
||||
int64_t hp = H / config.patch_size;
|
||||
|
||||
pos_vec = gen_2d_sincos_pos_embed(config.hidden_size, hp, wp,
|
||||
config.pos_embed_base_size, config.interpolation_scale);
|
||||
auto pos = ggml_new_tensor_3d(compute_ctx, GGML_TYPE_F32, config.hidden_size, wp * hp, 1);
|
||||
set_backend_tensor_data(pos, pos_vec.data());
|
||||
|
||||
auto runner_ctx = get_context();
|
||||
ggml_tensor* out = model.forward(&runner_ctx, x, timesteps, context, pos, context_mask);
|
||||
// learn_sigma: keep the noise prediction half of the output channels
|
||||
out = ggml_ext_slice(compute_ctx, out, 2, 0, config.in_channels);
|
||||
ggml_build_forward_expand(gf, out);
|
||||
return gf;
|
||||
}
|
||||
|
||||
sd::Tensor<float> compute(int n_threads,
|
||||
const sd::Tensor<float>& x,
|
||||
const sd::Tensor<float>& timesteps,
|
||||
const sd::Tensor<float>& context,
|
||||
const sd::Tensor<float>& context_mask) {
|
||||
auto get_graph = [&]() -> ggml_cgraph* {
|
||||
return build_graph(x, timesteps, context, context_mask);
|
||||
};
|
||||
return restore_trailing_singleton_dims(GGMLRunner::compute(get_graph, n_threads, false), x.dim());
|
||||
}
|
||||
|
||||
sd::Tensor<float> compute(int n_threads,
|
||||
const DiffusionParams& diffusion_params) override {
|
||||
GGML_ASSERT(diffusion_params.x != nullptr);
|
||||
GGML_ASSERT(diffusion_params.timesteps != nullptr);
|
||||
auto context = tensor_or_empty(diffusion_params.context);
|
||||
auto context_msk = tensor_or_empty(diffusion_params.y);
|
||||
return compute(n_threads,
|
||||
*diffusion_params.x,
|
||||
*diffusion_params.timesteps,
|
||||
context,
|
||||
context_msk);
|
||||
}
|
||||
};
|
||||
} // namespace PixArt
|
||||
|
||||
#endif // __SD_MODEL_DIFFUSION_PIXART_HPP__
|
||||
@@ -544,7 +544,7 @@ public:
|
||||
const String2TensorStorage& tensor_storage_map = {},
|
||||
const std::string& prefix = "")
|
||||
: version(version), decode_only(decode_only), use_video_decoder(use_video_decoder) {
|
||||
if (sd_version_is_dit(version)) {
|
||||
if (sd_version_is_dit(version) && version != VERSION_PIXART) {
|
||||
if (sd_version_uses_flux2_vae(version)) {
|
||||
dd_config.z_channels = 32;
|
||||
embed_dim = 32;
|
||||
@@ -678,7 +678,7 @@ struct AutoEncoderKL : public VAE {
|
||||
if (sd_version_is_sd1(version) || sd_version_is_sd2(version)) {
|
||||
scale_factor = 0.18215f;
|
||||
shift_factor = 0.f;
|
||||
} else if (sd_version_is_sdxl(version)) {
|
||||
} else if (sd_version_is_sdxl(version) || sd_version_is_pixart(version)) {
|
||||
scale_factor = 0.13025f;
|
||||
shift_factor = 0.f;
|
||||
} else if (sd_version_is_sd3(version)) {
|
||||
|
||||
@@ -556,12 +556,18 @@ namespace MiniMaxH3VAE {
|
||||
tensor.shape()[3]});
|
||||
}
|
||||
|
||||
static sd_tiling_params_t h3_tiling(sd_tiling_params_t params) {
|
||||
sd_tiling_params_t resolve_tiling_params(sd_tiling_params_t params) const override {
|
||||
if (!params.enabled) {
|
||||
params.target_overlap = 0.25f;
|
||||
}
|
||||
if (params.tile_size_w == 0 && params.rel_size_w == 0.f) {
|
||||
params.tile_size_w = 256;
|
||||
}
|
||||
if (params.tile_size_h == 0 && params.rel_size_h == 0.f) {
|
||||
params.tile_size_h = 256;
|
||||
}
|
||||
params.enabled = true;
|
||||
params.temporal_tiling = false;
|
||||
params.tile_size_x = 16;
|
||||
params.tile_size_y = 16;
|
||||
params.target_overlap = 0.25f;
|
||||
return params;
|
||||
}
|
||||
|
||||
@@ -605,7 +611,7 @@ namespace MiniMaxH3VAE {
|
||||
bool circular_x = false,
|
||||
bool circular_y = false) override {
|
||||
auto input = ensure_video_shape(x);
|
||||
auto tiling = h3_tiling(tiling_params);
|
||||
auto tiling = resolve_tiling_params(tiling_params);
|
||||
if (input.shape()[2] == 1) {
|
||||
auto encoded = VAE::encode(n_threads, input, tiling, circular_x, circular_y);
|
||||
if (!encoded.empty() && encoded.shape()[2] > 1) {
|
||||
@@ -646,7 +652,7 @@ namespace MiniMaxH3VAE {
|
||||
bool circular_y = false,
|
||||
bool silent = false) override {
|
||||
auto input = ensure_video_shape(x);
|
||||
auto tiling = h3_tiling(tiling_params);
|
||||
auto tiling = resolve_tiling_params(tiling_params);
|
||||
if (input.shape()[2] == 1) {
|
||||
auto decoded = VAE::decode(n_threads,
|
||||
input,
|
||||
|
||||
@@ -701,7 +701,7 @@ public:
|
||||
bool use_midblock_gn = false;
|
||||
taef2 = sd_version_uses_flux2_vae(version);
|
||||
|
||||
if (sd_version_is_dit(version)) {
|
||||
if (sd_version_is_dit(version) && !sd_version_is_pixart(version)) {
|
||||
z_channels = 16;
|
||||
}
|
||||
if (taef2) {
|
||||
|
||||
+82
-50
@@ -1,6 +1,9 @@
|
||||
#ifndef __SD_MODEL_VAE_VAE_HPP__
|
||||
#define __SD_MODEL_VAE_VAE_HPP__
|
||||
|
||||
#include <cmath>
|
||||
#include <limits>
|
||||
|
||||
#include "core/tensor_ggml.hpp"
|
||||
#include "model/common/block.hpp"
|
||||
#include "model/vae/vae_tiling.hpp"
|
||||
@@ -117,8 +120,8 @@ protected:
|
||||
int output_width,
|
||||
int output_height,
|
||||
int scale,
|
||||
int p_tile_size_x,
|
||||
int p_tile_size_y,
|
||||
int p_tile_size_w,
|
||||
int p_tile_size_h,
|
||||
float tile_overlap_factor,
|
||||
bool circular_x,
|
||||
bool circular_y,
|
||||
@@ -138,17 +141,28 @@ protected:
|
||||
}
|
||||
return output_tile;
|
||||
};
|
||||
return ::process_tiles_2d(input,
|
||||
output_width,
|
||||
output_height,
|
||||
scale,
|
||||
p_tile_size_x,
|
||||
p_tile_size_y,
|
||||
tile_overlap_factor,
|
||||
circular_x,
|
||||
circular_y,
|
||||
on_processing,
|
||||
silent);
|
||||
const bool original_circular_x = circular_x_enabled;
|
||||
const bool original_circular_y = circular_y_enabled;
|
||||
const int64_t latent_width = decode_graph ? input.shape()[0] : output_width;
|
||||
const int64_t latent_height = decode_graph ? input.shape()[1] : output_height;
|
||||
circular_x = circular_x || original_circular_x;
|
||||
circular_y = circular_y || original_circular_y;
|
||||
// Full-width axes wrap in convolutions; split axes wrap between tiles.
|
||||
set_circular_axes(circular_x && p_tile_size_w >= latent_width,
|
||||
circular_y && p_tile_size_h >= latent_height);
|
||||
auto output = ::process_tiles_2d(input,
|
||||
output_width,
|
||||
output_height,
|
||||
scale,
|
||||
p_tile_size_w,
|
||||
p_tile_size_h,
|
||||
tile_overlap_factor,
|
||||
circular_x && p_tile_size_w < latent_width,
|
||||
circular_y && p_tile_size_h < latent_height,
|
||||
on_processing,
|
||||
silent);
|
||||
set_circular_axes(original_circular_x, original_circular_y);
|
||||
return output;
|
||||
}
|
||||
|
||||
public:
|
||||
@@ -178,33 +192,48 @@ public:
|
||||
return supports_temporal_tiling(VAETemporalDirection::DECODE);
|
||||
}
|
||||
|
||||
void get_tile_sizes(int& tile_size_x,
|
||||
int& tile_size_y,
|
||||
virtual sd_tiling_params_t resolve_tiling_params(sd_tiling_params_t params) const {
|
||||
return params;
|
||||
}
|
||||
|
||||
bool get_tile_sizes(int& tile_size_w,
|
||||
int& tile_size_h,
|
||||
float& tile_overlap,
|
||||
const sd_tiling_params_t& params,
|
||||
int64_t latent_x,
|
||||
int64_t latent_y,
|
||||
float encoding_factor = 1.0f) {
|
||||
tile_overlap = std::max(std::min(params.target_overlap, 0.5f), 0.0f);
|
||||
auto get_tile_size = [&](int requested_size, float factor, int64_t latent_size) {
|
||||
const int default_tile_size = 32;
|
||||
const int min_tile_dimension = 4;
|
||||
int tile_size = default_tile_size;
|
||||
// factor <= 1 means simple fraction of the latent dimension
|
||||
// factor > 1 means number of tiles across that dimension
|
||||
if (factor > 0.f) {
|
||||
if (factor > 1.0)
|
||||
factor = 1 / (factor - factor * tile_overlap + tile_overlap);
|
||||
tile_size = static_cast<int>(std::round(latent_size * factor));
|
||||
} else if (requested_size >= min_tile_dimension) {
|
||||
tile_size = requested_size;
|
||||
int64_t latent_w,
|
||||
int64_t latent_h) {
|
||||
const auto tiling = resolve_tiling_params(params);
|
||||
if (latent_w <= 0 || latent_h <= 0 ||
|
||||
latent_w > std::numeric_limits<int>::max() || latent_h > std::numeric_limits<int>::max() ||
|
||||
!std::isfinite(tiling.target_overlap)) {
|
||||
LOG_ERROR("invalid VAE tiling dimensions or overlap");
|
||||
return false;
|
||||
}
|
||||
const int scale_factor = get_scale_factor();
|
||||
tile_overlap = std::max(std::min(tiling.target_overlap, 0.5f), 0.0f);
|
||||
auto get_tile_size = [&](int requested_size, double factor, int64_t latent_size, int& tile_size) {
|
||||
if (requested_size < 0 || !std::isfinite(factor) || factor < 0.0) {
|
||||
LOG_ERROR("VAE tile sizes and relative sizes must be finite and non-negative");
|
||||
return false;
|
||||
}
|
||||
tile_size = static_cast<int>(tile_size * encoding_factor);
|
||||
return std::max(std::min(tile_size, static_cast<int>(latent_size)), min_tile_dimension);
|
||||
const int min_tile_dimension = std::min(4, static_cast<int>(latent_size));
|
||||
double size = (requested_size > 0 ? requested_size : 256) / scale_factor;
|
||||
if (factor > 0.0) {
|
||||
if (factor > 1.0) {
|
||||
factor = 1.0 / (factor * (1.0 - tile_overlap) + tile_overlap);
|
||||
}
|
||||
size = std::floor(static_cast<double>(latent_size) * factor);
|
||||
}
|
||||
if (size < min_tile_dimension && (requested_size > 0 || factor > 0.0)) {
|
||||
LOG_ERROR("VAE tile size must be at least %d image pixels on this axis", min_tile_dimension * scale_factor);
|
||||
return false;
|
||||
}
|
||||
tile_size = static_cast<int>(std::min(static_cast<double>(latent_size), std::max<double>(min_tile_dimension, size)));
|
||||
return true;
|
||||
};
|
||||
|
||||
tile_size_x = get_tile_size(params.tile_size_x, params.rel_size_x, latent_x);
|
||||
tile_size_y = get_tile_size(params.tile_size_y, params.rel_size_y, latent_y);
|
||||
return get_tile_size(tiling.tile_size_w, tiling.rel_size_w, latent_w, tile_size_w) &&
|
||||
get_tile_size(tiling.tile_size_h, tiling.rel_size_h, latent_h, tile_size_h);
|
||||
}
|
||||
|
||||
virtual sd::Tensor<float> encode(int n_threads,
|
||||
@@ -213,6 +242,7 @@ public:
|
||||
bool circular_x = false,
|
||||
bool circular_y = false) {
|
||||
int64_t t0 = ggml_time_ms();
|
||||
tiling_params = resolve_tiling_params(tiling_params);
|
||||
sd::Tensor<float> input = x;
|
||||
sd::Tensor<float> output;
|
||||
if (scale_input) {
|
||||
@@ -224,21 +254,19 @@ public:
|
||||
int64_t W = input.shape()[0] / scale_factor;
|
||||
int64_t H = input.shape()[1] / scale_factor;
|
||||
float tile_overlap;
|
||||
int tile_size_x, tile_size_y;
|
||||
// Image VAE encode is more sensitive to tile boundary context than decode.
|
||||
// Keep the smaller legacy factor for video VAEs, but default image encode
|
||||
// tiles to 64 latent pixels so a 512px SD image is encoded as one tile.
|
||||
const float encode_tile_factor = sd_version_is_minimax_h3(version) ? 1.f : (sd_version_is_wan(version) || sd_version_is_hunyuan_video(version) || sd_version_is_ltxav(version)) ? 1.30539f
|
||||
: 2.0f;
|
||||
get_tile_sizes(tile_size_x, tile_size_y, tile_overlap, tiling_params, W, H, encode_tile_factor);
|
||||
LOG_VERBOSE("VAE Tile size: %dx%d", tile_size_x, tile_size_y);
|
||||
int tile_size_w, tile_size_h;
|
||||
if (!get_tile_sizes(tile_size_w, tile_size_h, tile_overlap, tiling_params, W, H)) {
|
||||
return {};
|
||||
}
|
||||
LOG_VERBOSE("VAE encode tile size: %dx%d pixels (%dx%d latent)",
|
||||
tile_size_w * scale_factor, tile_size_h * scale_factor, tile_size_w, tile_size_h);
|
||||
output = tiled_compute(input,
|
||||
n_threads,
|
||||
static_cast<int>(W),
|
||||
static_cast<int>(H),
|
||||
scale_factor,
|
||||
tile_size_x,
|
||||
tile_size_y,
|
||||
tile_size_w,
|
||||
tile_size_h,
|
||||
tile_overlap,
|
||||
circular_x,
|
||||
circular_y,
|
||||
@@ -271,6 +299,7 @@ public:
|
||||
bool circular_y = false,
|
||||
bool silent = false) {
|
||||
int64_t t0 = ggml_time_ms();
|
||||
tiling_params = resolve_tiling_params(tiling_params);
|
||||
sd::Tensor<float> input = x;
|
||||
sd::Tensor<float> output;
|
||||
|
||||
@@ -279,10 +308,13 @@ public:
|
||||
int64_t W = input.shape()[0] * scale_factor;
|
||||
int64_t H = input.shape()[1] * scale_factor;
|
||||
float tile_overlap;
|
||||
int tile_size_x, tile_size_y;
|
||||
get_tile_sizes(tile_size_x, tile_size_y, tile_overlap, tiling_params, input.shape()[0], input.shape()[1]);
|
||||
int tile_size_w, tile_size_h;
|
||||
if (!get_tile_sizes(tile_size_w, tile_size_h, tile_overlap, tiling_params, input.shape()[0], input.shape()[1])) {
|
||||
return {};
|
||||
}
|
||||
if (!silent) {
|
||||
LOG_VERBOSE("VAE Tile size: %dx%d", tile_size_x, tile_size_y);
|
||||
LOG_VERBOSE("VAE decode tile size: %dx%d pixels (%dx%d latent)",
|
||||
tile_size_w * scale_factor, tile_size_h * scale_factor, tile_size_w, tile_size_h);
|
||||
}
|
||||
output = tiled_compute(
|
||||
input,
|
||||
@@ -290,8 +322,8 @@ public:
|
||||
static_cast<int>(W),
|
||||
static_cast<int>(H),
|
||||
scale_factor,
|
||||
tile_size_x,
|
||||
tile_size_y,
|
||||
tile_size_w,
|
||||
tile_size_h,
|
||||
tile_overlap,
|
||||
circular_x,
|
||||
circular_y,
|
||||
|
||||
+14
-1
@@ -322,7 +322,13 @@ bool ModelLoader::init_from_safetensors_index_file(const std::string& file_path,
|
||||
}
|
||||
|
||||
for (const std::string& shard_path : shard_paths) {
|
||||
if (!parse_file(shard_path, prefix)) {
|
||||
FileStamp stamp;
|
||||
if (!read_file_stamp(shard_path, stamp)) {
|
||||
return false;
|
||||
}
|
||||
parsed_dependencies_.push_back(stamp);
|
||||
LOG_INFO("load %s using safetensors format", shard_path.c_str());
|
||||
if (!init_from_safetensors_file(shard_path, prefix)) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -527,6 +533,10 @@ SDVersion ModelLoader::get_sd_version() const {
|
||||
return VERSION_ERNIE_IMAGE;
|
||||
}
|
||||
if (tensor_storage.name.find("model.diffusion_model.adaln_single.emb.timestep_embedder.linear_1.bias") != std::string::npos) {
|
||||
// PixArt shares this signature with LTX-AV; pos_embed.proj is PixArt-only.
|
||||
if (tensor_storage_map.find("model.diffusion_model.pos_embed.proj.weight") != tensor_storage_map.end()) {
|
||||
return VERSION_PIXART;
|
||||
}
|
||||
return VERSION_LTXAV;
|
||||
}
|
||||
if (tensor_storage.name.find("model.diffusion_model.video_patch_proj.weight") != std::string::npos &&
|
||||
@@ -1591,6 +1601,9 @@ bool ModelLoader::tensor_should_be_converted(const TensorStorage& tensor_storage
|
||||
// Pass, do not convert. For Unet
|
||||
} else if (contains(name, "embedding")) {
|
||||
// Pass, do not convert embedding
|
||||
} else if (contains(name, "scale_shift_table")) {
|
||||
// Pass, do not convert. adaLN modulation tables (PixArt, LTXV) are sliced
|
||||
// element-wise, which is invalid on quantized block layouts.
|
||||
} else if (ends_with(name, "_pad_token")) {
|
||||
// Pass, do not convert. LLaDA-Image stores its pad tokens far outside the f16
|
||||
// range, so any format with an f16 scale or payload turns them into inf.
|
||||
|
||||
@@ -105,6 +105,7 @@ const char* model_version_to_str[] = {
|
||||
"SenseNova U1.5",
|
||||
"LLaDA-Image",
|
||||
"ESRGAN",
|
||||
"PixArt",
|
||||
};
|
||||
|
||||
static_assert(VERSION_COUNT == sizeof(model_version_to_str) / sizeof(model_version_to_str[0]),
|
||||
@@ -125,6 +126,18 @@ void calculate_alphas_cumprod(float* alphas_cumprod,
|
||||
}
|
||||
}
|
||||
|
||||
void calculate_alphas_cumprod_linear_beta(float* alphas_cumprod,
|
||||
float beta_start,
|
||||
float beta_end,
|
||||
int timesteps = TIMESTEPS) {
|
||||
float product = 1.0f;
|
||||
for (int i = 0; i < timesteps; i++) {
|
||||
float beta = beta_start + (beta_end - beta_start) * ((float)i / (timesteps - 1));
|
||||
product *= 1.0f - beta;
|
||||
alphas_cumprod[i] = product;
|
||||
}
|
||||
}
|
||||
|
||||
template <typename T, typename = void>
|
||||
struct has_set_runtime_backends : std::false_type {};
|
||||
template <typename T>
|
||||
@@ -666,6 +679,10 @@ void StableDiffusionGGML::refresh_compvis_denoiser_sigmas() {
|
||||
std::vector<float> alphas_cumprod(TIMESTEPS);
|
||||
if (file_alphas_cumprod.size() == TIMESTEPS) {
|
||||
alphas_cumprod = file_alphas_cumprod;
|
||||
} else if (sd_version_is_pixart(version)) {
|
||||
// PixArt checkpoints train with a linear beta schedule (0.0001 -> 0.02)
|
||||
// instead of the scaled_linear schedule used by SD1.x/SDXL.
|
||||
calculate_alphas_cumprod_linear_beta(alphas_cumprod.data(), 0.0001f, 0.02f);
|
||||
} else {
|
||||
calculate_alphas_cumprod(alphas_cumprod.data());
|
||||
}
|
||||
@@ -2731,7 +2748,7 @@ int StableDiffusionGGML::get_diffusion_model_down_factor() {
|
||||
if (sd_version_is_dit(version)) {
|
||||
if (sd_version_is_sensenova_u1(version)) {
|
||||
down_factor = 32;
|
||||
} else if (version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_wan(version) || sd_version_is_lingbot_video(version) || sd_version_is_minimax_h3(version)) {
|
||||
} else if (version == VERSION_QWEN_IMAGE_2_1 || sd_version_is_wan(version) || sd_version_is_lingbot_video(version) || sd_version_is_minimax_h3(version) || sd_version_is_pixart(version)) {
|
||||
down_factor = 2;
|
||||
} else {
|
||||
down_factor = 1;
|
||||
@@ -2769,6 +2786,8 @@ int StableDiffusionGGML::get_latent_channel() {
|
||||
latent_channel = 128;
|
||||
} else if (sd_version_is_mage_flow(version)) {
|
||||
latent_channel = 128;
|
||||
} else if (sd_version_is_pixart(version)) {
|
||||
latent_channel = 4;
|
||||
} else {
|
||||
latent_channel = 16;
|
||||
}
|
||||
@@ -2864,14 +2883,26 @@ sd::Tensor<float> StableDiffusionGGML::decode_first_stage(const sd::Tensor<float
|
||||
return sd::ops::clamp((x + 1.f) * 0.5f, 0.0f, 1.0f);
|
||||
}
|
||||
auto latents = first_stage_model->diffusion_to_vae_latents(x);
|
||||
auto decoded = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
|
||||
const bool prefer_temporal_tiling = decode_video && first_stage_model->can_temporal_tile_decode();
|
||||
while (decoded.empty() &&
|
||||
sd::backend_fit::prepare_vae_decode_retry_tiling(vae_tiling_params, prefer_temporal_tiling,
|
||||
first_stage_model->last_compute_status())) {
|
||||
decoded = first_stage_model->decode(n_threads, latents, vae_tiling_params, decode_video, circular_x, circular_y);
|
||||
auto tiling_params = first_stage_model->resolve_tiling_params(vae_tiling_params);
|
||||
const bool prefer_temporal_tiling = decode_video && latents.dim() == 5 && latents.shape()[2] > 1 &&
|
||||
first_stage_model->can_temporal_tile_decode();
|
||||
for (;;) {
|
||||
int tile_size_w = static_cast<int>(latents.shape()[0]);
|
||||
int tile_size_h = static_cast<int>(latents.shape()[1]);
|
||||
float tile_overlap;
|
||||
if (tiling_params.enabled &&
|
||||
!first_stage_model->get_tile_sizes(tile_size_w, tile_size_h, tile_overlap, tiling_params,
|
||||
latents.shape()[0], latents.shape()[1])) {
|
||||
return {};
|
||||
}
|
||||
auto decoded = first_stage_model->decode(n_threads, latents, tiling_params, decode_video, circular_x, circular_y);
|
||||
if (!decoded.empty() ||
|
||||
!sd::backend_fit::prepare_vae_decode_retry_tiling(tiling_params, prefer_temporal_tiling,
|
||||
first_stage_model->last_compute_status(),
|
||||
tile_size_w, tile_size_h, first_stage_model->get_scale_factor())) {
|
||||
return decoded;
|
||||
}
|
||||
}
|
||||
return decoded;
|
||||
}
|
||||
|
||||
sd::Tensor<float> StableDiffusionGGML::normalize_ltx_video_latents(const sd::Tensor<float>& x) {
|
||||
|
||||
+15
-13
@@ -35,19 +35,21 @@ namespace sd::pipeline {
|
||||
return original_axes;
|
||||
}
|
||||
|
||||
int tile_size_x, tile_size_y;
|
||||
int tile_size_w, tile_size_h;
|
||||
float overlap;
|
||||
int latent_size_x = request.width / request.vae_scale_factor;
|
||||
int latent_size_y = request.height / request.vae_scale_factor;
|
||||
sd->first_stage_model->get_tile_sizes(tile_size_x,
|
||||
tile_size_y,
|
||||
overlap,
|
||||
sd_img_gen_params->vae_tiling_params,
|
||||
latent_size_x,
|
||||
latent_size_y);
|
||||
int latent_size_w = request.width / request.vae_scale_factor;
|
||||
int latent_size_h = request.height / request.vae_scale_factor;
|
||||
if (!sd->first_stage_model->get_tile_sizes(tile_size_w,
|
||||
tile_size_h,
|
||||
overlap,
|
||||
sd_img_gen_params->vae_tiling_params,
|
||||
latent_size_w,
|
||||
latent_size_h)) {
|
||||
return original_axes;
|
||||
}
|
||||
|
||||
sd->circular_x = sd->circular_x && (tile_size_x >= latent_size_x);
|
||||
sd->circular_y = sd->circular_y && (tile_size_y >= latent_size_y);
|
||||
sd->circular_x = sd->circular_x && (tile_size_w >= latent_size_w);
|
||||
sd->circular_y = sd->circular_y && (tile_size_h >= latent_size_h);
|
||||
|
||||
if (sd->first_stage_model) {
|
||||
sd->first_stage_model->set_circular_axes(sd->circular_x, sd->circular_y);
|
||||
@@ -56,8 +58,8 @@ namespace sd::pipeline {
|
||||
sd->preview_vae->set_circular_axes(sd->circular_x, sd->circular_y);
|
||||
}
|
||||
|
||||
sd->circular_x = original_axes.circular_x && (tile_size_x < latent_size_x);
|
||||
sd->circular_y = original_axes.circular_y && (tile_size_y < latent_size_y);
|
||||
sd->circular_x = original_axes.circular_x && (tile_size_w < latent_size_w);
|
||||
sd->circular_y = original_axes.circular_y && (tile_size_h < latent_size_h);
|
||||
|
||||
return original_axes;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#include "model_builders.h"
|
||||
|
||||
#include <cmath>
|
||||
#include <cstring>
|
||||
#include <utility>
|
||||
|
||||
@@ -28,6 +29,7 @@
|
||||
#include "model/diffusion/mmdit.hpp"
|
||||
#include "model/diffusion/model.hpp"
|
||||
#include "model/diffusion/pid.hpp"
|
||||
#include "model/diffusion/pixart.hpp"
|
||||
#include "model/diffusion/qwen_image.hpp"
|
||||
#include "model/diffusion/qwen_image_2_1.hpp"
|
||||
#include "model/diffusion/sensenova_u1.h"
|
||||
@@ -301,6 +303,19 @@ namespace sd::model_builders {
|
||||
weight_manager,
|
||||
sd_ctx_params->model_args);
|
||||
}
|
||||
} else if (version == VERSION_PIXART) {
|
||||
result.conditioner = std::make_shared<T5CLIPEmbedder>(ctx.backends.runtime_backend(SDBackendModule::TE),
|
||||
tensor_storage_map,
|
||||
true,
|
||||
0,
|
||||
false,
|
||||
weight_manager,
|
||||
sd_ctx_params->model_args);
|
||||
result.diffusion = std::make_shared<PixArt::PixArtRunner>(ctx.backends.runtime_backend(SDBackendModule::DIFFUSION),
|
||||
tensor_storage_map,
|
||||
"model.diffusion_model",
|
||||
weight_manager,
|
||||
sd_ctx_params->model_args);
|
||||
} else if (sd_version_is_mage_flow(version)) {
|
||||
result.conditioner = std::make_shared<LLMEmbedder>(ctx.backends.runtime_backend(SDBackendModule::TE),
|
||||
tensor_storage_map,
|
||||
@@ -562,6 +577,23 @@ namespace sd::model_builders {
|
||||
false,
|
||||
vae_version,
|
||||
weight_manager);
|
||||
if (sd_version_is_pixart(version)) {
|
||||
// Alpha-512 and Sigma share tensor layouts; Alpha-512 needs an explicit scale override.
|
||||
if (tensor_storage_map.count("model.diffusion_model.adaln_single.emb.resolution_embedder.linear_1.weight") != 0) {
|
||||
model->scale_factor = 0.18215f;
|
||||
}
|
||||
for (const auto& [key, value] : parse_key_value_args(sd_ctx_params->model_args, "model arg")) {
|
||||
if (key == "pixart_vae_scale_factor") {
|
||||
float parsed = 0.f;
|
||||
if (parse_strict_float(value, parsed) && std::isfinite(parsed) && parsed > 0.f) {
|
||||
model->scale_factor = parsed;
|
||||
} else {
|
||||
LOG_WARN("ignoring invalid PixArt model arg '%s=%s'", key.c_str(), value.c_str());
|
||||
}
|
||||
}
|
||||
}
|
||||
LOG_VERBOSE("pixart: VAE scale factor = %.5f", model->scale_factor);
|
||||
}
|
||||
if (sd_version_is_sdxl(version) &&
|
||||
(strlen(SAFE_STR(sd_ctx_params->vae_path)) == 0 || sd_ctx_params->force_sdxl_vae_conv_scale || options.external_vae_is_invalid)) {
|
||||
float vae_conv_2d_scale = 1.f / 32.f;
|
||||
|
||||
+24
-24
@@ -142,8 +142,8 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
int output_width,
|
||||
int output_height,
|
||||
int scale,
|
||||
int p_tile_size_x,
|
||||
int p_tile_size_y,
|
||||
int p_tile_size_w,
|
||||
int p_tile_size_h,
|
||||
float tile_overlap_factor,
|
||||
bool circular_x,
|
||||
bool circular_y,
|
||||
@@ -168,28 +168,28 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
|
||||
int num_tiles_x;
|
||||
float tile_overlap_factor_x;
|
||||
sd_tiling_calc_tiles(num_tiles_x, tile_overlap_factor_x, small_width, p_tile_size_x, tile_overlap_factor, circular_x);
|
||||
sd_tiling_calc_tiles(num_tiles_x, tile_overlap_factor_x, small_width, p_tile_size_w, tile_overlap_factor, circular_x);
|
||||
|
||||
int num_tiles_y;
|
||||
float tile_overlap_factor_y;
|
||||
sd_tiling_calc_tiles(num_tiles_y, tile_overlap_factor_y, small_height, p_tile_size_y, tile_overlap_factor, circular_y);
|
||||
sd_tiling_calc_tiles(num_tiles_y, tile_overlap_factor_y, small_height, p_tile_size_h, tile_overlap_factor, circular_y);
|
||||
|
||||
int tile_overlap_x = static_cast<int32_t>(p_tile_size_x * tile_overlap_factor_x);
|
||||
int non_tile_overlap_x = p_tile_size_x - tile_overlap_x;
|
||||
int tile_overlap_y = static_cast<int32_t>(p_tile_size_y * tile_overlap_factor_y);
|
||||
int non_tile_overlap_y = p_tile_size_y - tile_overlap_y;
|
||||
int tile_size_x = p_tile_size_x < small_width ? p_tile_size_x : small_width;
|
||||
int tile_size_y = p_tile_size_y < small_height ? p_tile_size_y : small_height;
|
||||
int input_tile_size_x = tile_size_x;
|
||||
int input_tile_size_y = tile_size_y;
|
||||
int output_tile_size_x = tile_size_x;
|
||||
int output_tile_size_y = tile_size_y;
|
||||
int tile_overlap_x = static_cast<int32_t>(p_tile_size_w * tile_overlap_factor_x);
|
||||
int non_tile_overlap_x = p_tile_size_w - tile_overlap_x;
|
||||
int tile_overlap_y = static_cast<int32_t>(p_tile_size_h * tile_overlap_factor_y);
|
||||
int non_tile_overlap_y = p_tile_size_h - tile_overlap_y;
|
||||
int tile_size_w = p_tile_size_w < small_width ? p_tile_size_w : small_width;
|
||||
int tile_size_h = p_tile_size_h < small_height ? p_tile_size_h : small_height;
|
||||
int input_tile_size_w = tile_size_w;
|
||||
int input_tile_size_h = tile_size_h;
|
||||
int output_tile_size_w = tile_size_w;
|
||||
int output_tile_size_h = tile_size_h;
|
||||
if (decode) {
|
||||
output_tile_size_x *= scale;
|
||||
output_tile_size_y *= scale;
|
||||
output_tile_size_w *= scale;
|
||||
output_tile_size_h *= scale;
|
||||
} else {
|
||||
input_tile_size_x *= scale;
|
||||
input_tile_size_y *= scale;
|
||||
input_tile_size_w *= scale;
|
||||
input_tile_size_h *= scale;
|
||||
}
|
||||
|
||||
int num_tiles = num_tiles_x * num_tiles_y;
|
||||
@@ -205,9 +205,9 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
}
|
||||
for (int y = 0; y < small_height && !last_y; y += non_tile_overlap_y) {
|
||||
int dy = 0;
|
||||
if (!circular_y && y + tile_size_y >= small_height) {
|
||||
if (!circular_y && y + tile_size_h >= small_height) {
|
||||
int original_y = y;
|
||||
y = small_height - tile_size_y;
|
||||
y = small_height - tile_size_h;
|
||||
dy = original_y - y;
|
||||
if (decode) {
|
||||
dy *= scale;
|
||||
@@ -216,9 +216,9 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
}
|
||||
for (int x = 0; x < small_width && !last_x; x += non_tile_overlap_x) {
|
||||
int dx = 0;
|
||||
if (!circular_x && x + tile_size_x >= small_width) {
|
||||
if (!circular_x && x + tile_size_w >= small_width) {
|
||||
int original_x = x;
|
||||
x = small_width - tile_size_x;
|
||||
x = small_width - tile_size_w;
|
||||
dx = original_x - x;
|
||||
if (decode) {
|
||||
dx *= scale;
|
||||
@@ -235,12 +235,12 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
int overlap_y_out = decode ? tile_overlap_y * scale : tile_overlap_y;
|
||||
|
||||
int64_t t1 = ggml_time_ms();
|
||||
auto input_tile = sd_tensor_split_2d(input, input_tile_size_x, input_tile_size_y, x_in, y_in);
|
||||
auto input_tile = sd_tensor_split_2d(input, input_tile_size_w, input_tile_size_h, x_in, y_in);
|
||||
auto output_tile = on_processing(input_tile);
|
||||
if (output_tile.empty()) {
|
||||
return {};
|
||||
}
|
||||
GGML_ASSERT(output_tile.shape()[0] == output_tile_size_x && output_tile.shape()[1] == output_tile_size_y);
|
||||
GGML_ASSERT(output_tile.shape()[0] == output_tile_size_w && output_tile.shape()[1] == output_tile_size_h);
|
||||
if (output.empty()) {
|
||||
std::vector<int64_t> output_shape = output_tile.shape();
|
||||
output_shape[0] = output_width;
|
||||
|
||||
@@ -11,8 +11,8 @@ sd::Tensor<float> process_tiles_2d(const sd::Tensor<float>& input,
|
||||
int output_width,
|
||||
int output_height,
|
||||
int scale,
|
||||
int p_tile_size_x,
|
||||
int p_tile_size_y,
|
||||
int p_tile_size_w,
|
||||
int p_tile_size_h,
|
||||
float tile_overlap_factor,
|
||||
bool circular_x,
|
||||
bool circular_y,
|
||||
|
||||
@@ -295,6 +295,12 @@ bool T5UniGramTokenizer::encode(const std::string& input, std::vector<int>& resu
|
||||
std::vector<int32_t> tokens;
|
||||
std::vector<std::string> token_strs;
|
||||
std::string normalized = normalize(input);
|
||||
if (normalized.empty()) {
|
||||
// HF reference tokenizers emit no pieces for empty input; pad_tokens
|
||||
// still appends EOS so the sequence becomes [EOS] + padding.
|
||||
result = std::move(tokens);
|
||||
return true;
|
||||
}
|
||||
auto splited_texts = split_with_special_tokens(normalized, special_tokens);
|
||||
if (splited_texts.empty()) {
|
||||
splited_texts.push_back(normalized); // for empty string
|
||||
|
||||
Reference in New Issue
Block a user