llama.cpp

upstream-archive/llama.cpp

Fork 0

mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-27 08:27:30 -05:00

Commit Graph

Author	SHA1	Message	Date
Georgi Gerganov	5b59b83f4e	metal : add MoE and SSM_CONV fusion optimizations (#28948 ) * metal : add top-k MoE fusion Adds a Metal fusion for SOFT_MAX + ARGSORT + GET_ROWS with optional routing-weight normalization and scale, matching the top-k MoE fusion available in the CUDA and Vulkan backends. The fused kernel writes the selected expert ids and routing weights directly, eliding the separate softmax, argsort, get-rows, sum-rows, clamp, div and scale kernels. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add MoE weighted reduction fusion Fuses MUL(experts, weights) plus the expert VIEW/ADD chain into one kernel that computes the weighted sum directly. The graph_optimize hook keeps the expert and weight buffers alive until the fused output so the allocator cannot reuse them while the kernel is still reading them. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * tests : expose MoE weighted reduction in fusion baseline Use 2 experts per token in the generated MoE test models so the Metal MoE weighted reduction fusion (MUL + ADD) is exercised by test-fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : fuse RMS_NORM + SCALE Adds NORM/RMS_NORM + SCALE fusion to the Metal backend by reusing the norm+mul kernel with a scalar scale flag. Adds test coverage for both NORM+SCALE and RMS_NORM+SCALE and regenerates the fusion baseline. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constant for RMS_NORM + SCALE Replaces the runtime use_scale karg with a Metal function constant. The norm+mul kernel is compiled with FC_norm_use_scale=false for MUL fusion and FC_norm_use_scale=true for SCALE fusion, so the fused kernel has no runtime branch. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constant for top-k MoE with_norm Replaces the runtime with_norm karg with a Metal function constant. The top-k MoE kernel is compiled separately for the normalized and non-normalized routing variants, removing the runtime branch. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename moe_weighted_reduction suffix to moe_reduce Shortens the MoE weighted-reduction fusion identifiers, kernel, pipeline, matcher, args struct, and test op name from moe_weighted_reduction to moe_reduce. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion Adds dense mat-vec activation fusion for sigmoid/silu and bias+softplus. The mat-vec kernels apply the activation/bias epilogue via function constants, avoiding the separate unary/add passes. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : revert MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion The mat-vec activation fusion regressed decode throughput on Qwen3.6-35B-A3B by ~8% (tg32 81.5 vs 88.5 t/s). The regression is caused by loss of concurrency: the standalone unary kernels previously overlapped with other mat-vec work, while fusing the activation into the mat-vec kernel serializes it on the critical path. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add SSM_CONV + UNARY (silu) fusion The SSM_CONV kernels apply silu directly via a function constant, eliding the separate unary pass. Regenerates the fusion baseline. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : address fusion review comments - Fix declaration/table alignment - Rename top-k MoE kargs fields to val_clamp / val_scale - Move moe-reduce alloc-deps handling into a general fusion helper - Remove the public moe-reduce matcher API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : fix unused parameter in top-k MoE fusion check Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : guard SSM_CONV fusion lookup behind use_fusion Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : track all fused outputs in graph reorder Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : keep top-k MoE logits alive until fused output Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : refactor alloc deps to pattern-driven approach Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : check fused kernel destination in concurrency tracking Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * meta : forward graph_optimize to underlying backends Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use vector for fusion table Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * meta : keep graph_optimize unimplemented Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * parallel : fix non-deterministic prompt selection Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * parallel : support dummy models and add global logits run hash Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : sync cross-device copies with destination completion event Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : avoid const_cast in fusion alloc deps Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : skip fusions with aliased sources Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : hide fusion pattern definition Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use vector fusion op sequences Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : drop redundant struct keywords Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : add alloc deps comment separator Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : generalize fusion output memory ranges Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename fusion out_offsets to outs Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : avoid dst vector in memory range check Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : optimize fusion matching and multi-output handling - use pointer arithmetic for fusion info count lookup - avoid heap allocations in top-k MoE and MoE reduce pattern matchers - use fusion outs for multi-output subgraph checks Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * Revert "parallel : support dummy models and add global logits run hash" This reverts commit 57c7caf941c1b43c270fd5009c9f175063522e96. * fusion : update MTL.csv * metal : unroll constant loops in top-k MoE kernel Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constants for top-k MoE n_expert and top_k Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : rename fusion kargs to scale and clamp Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use function constants for moe_reduce and ssm_conv Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * fusion : update MTL.csv	2026-09-19 13:14:44 +03:00
Georgi Gerganov	bbdd9f246e	tests : add fusion baseline README and broaden fusion CI triggers (#28893 ) * tests : add README for updating the per-backend fusion baselines Assisted-by: pi:llama.cpp/Qwen3.8-27B * ci : trigger fusion on changes to test-llama-archs.cpp and src/models the dummy models and their architectures drive the fusion baselines, so a change to either can alter the per-fusion counters and should re-run the fusion job. Assisted-by: pi:llama.cpp/Qwen3.8-27B * tests : merge the fusion build commands in the README assisted-by: pi:llama.cpp/Qwen3.8-27B * pi : require explicit permission before posting PR/issue comments assisted-by: pi:llama.cpp/Qwen3.8-27B	2026-09-14 15:45:05 +03:00
Georgi Gerganov	a2878d30df	metal : single-source fusion table + fusion debug rework (#28164 ) * metal : rework fusion patterns into a single table All fusable op patterns for the Metal backend are now declared once in a fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer (ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next, compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext for the structural checks, and differ only in the mode used for the pattern check (STRUCTURAL at optimize time, since tensors are not allocated yet, and FULL at compute time, including Metal buffer placement). This also protects the snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph optimization, which was previously unprotected. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fix absolute output indices in fusion patterns ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query was passing a relative index (n_ops - 1). As a result the last node of every pattern was not recognized as an output and was subjected to the elidable use-count check, which failed for essentially all fusions. This silently disabled the norm/MUL fusion and caused a ~5% token-generation regression. Pass the absolute graph index of the last node instead. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : fuse gated_delta_net with cache cpy Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net kernel is followed by a cpy that scatters its recurrent state snapshots into the KV cache, the kernel writes the snapshots straight into the cache buffer and the trailing cpy is elided. The gdn output has other consumers (the attn scores view), so unlike the elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks, making the pattern-specific check callback the sole validator. Packing (ggml_metal_fuse_max) now matches on the same view-transparent node sequence that the compute phase uses, so the gdn + cache cpy group is packed along with any intermediate views and stays adjacent through the reorder. The fused cpy is a view consumer of the gdn (it writes the cache directly), so its mem-range is skipped in the encoder; the skip is restricted to CPY nodes consuming the previous fused node through a view so other fusions are unaffected. Add test_gated_delta_net_cache_fusion and register 5 cases. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : drop is_view_consumer mem-range skip The is_view_consumer skip was carried over from the upstream gated_delta_net cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in the concurrency tracker only ever adds a (conservative) memory barrier at the fusion point. It can never remove a barrier, so it cannot introduce a race. The worst case is one spurious barrier per gdn+cache-cpy fusion, which is within run-to-run noise on Qwen3.5-0.8B Q8_0. Dropping the check keeps the mem-range loop uniform for all fused groups. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename gated_delta_net fused state output args Rename the fused cache-write kernel argument to match the rest of the kargs: state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id bid_state_out -> bid_out. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : rename raw fusion flag to unsafe raw did not convey that the flag opts a fusion pattern out of the generic elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks). rename it to 'unsafe' to make explicit that the pattern's check callback is the sole validator and must re-establish the safety guarantees itself. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : tidy fusion pattern checks and table - const-correct ggml_metal_fuse_outputs buffer - annotate unused check-callback parameters - drop a redundant size_t cast - align the ops/table initializers and add blank-line separation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * metal : add generic fusion stats via ad-hoc proc-address API Add a device-owned fusion context that lets a test tool count how many times each fusion pattern fires and toggle fusion. It is exposed through the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names so the testing tool is backend-agnostic: - ggml_backend_fusion_stats_init: start collecting fusion stats; when a context is created afterwards it registers the labels/counters and encodes single-threaded (n_cb == 0) so the counters are race-free - ggml_backend_fusion_stats_reset / _get_stats / _set_enabled The context lives on the metal device (not on the last backend context), so counters accumulate across contexts and reads are always consistent. The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE and can be overridden by the test through set_enabled. Labels are synthesized from the fuse table via ggml_metal_fuse_label (e.g. "GATED_DELTA_NET+CPY"). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add fusion count regression test with per-backend baseline test-fusion runs every dummy model generated by test-llama-archs on a single backend (single-threaded encoding, n_cb == 0) with fusion enabled and disabled, and for each mode (prefill / decode) reports the per-fusion counters and the NMSE between the fused and unfused logits, plus the NMSE against a CPU reference. A fusion pattern that silently stops matching (or fires when it should not) is caught as a regression by comparing the counters against a committed per-backend TSV baseline: - --record writes the golden baseline, --check (default) validates it - the unfused run doubles as a control: its counters must be all-zero - NMSE is skipped when it is NaN or the arch is already broken on the device (e.g. plamo2 on Metal), so the count check is the hard gate - baseline counts depend only on graph structure, not weights (verified stable across weight seeds) - the fusion stats API is resolved through the ad-hoc get_proc_address mechanism with generic names; a backend that does not export it makes the test fail with an error The committed MTL0.tsv baseline covers 110 dummy archs (298 rows). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename fusion api helpers to match stats_init signature Align the test with the ad-hoc fusion stats API: fusion_stats_init no longer takes an enable bool (stats are turned on by calling it), so the proc-address wrappers and typedefs are renamed to the api_* convention. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : rename backend to device in fusion test CLI The fusion test operates on a compute device (e.g. MTL0), not a backend, so rename the --backend argument to --device and the backend_name variable to device_name. Keep "backend" where it refers to the ggml backend interface (the ad-hoc proc-address mechanism). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : add --model and --help to fusion test --model FILE runs the fusion regression test over a single model file instead of enumerating a --models DIR. --models and --model are mutually exclusive. Also add a --help/-h option that prints the usage. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : use backend base name for fusion baseline output The fusion test is invoked with a specific device name (e.g. MTL0), but its output - the recorded baseline and the header it writes - should be named after the backend base name (e.g. MTL, via ggml_backend_reg_name), since the counters depend on the backend, not on the specific device index. Rename the committed baseline to MTL.tsv. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : run fusion test from ci instead of ctest The fusion test needs Metal and generates a lot of dummy models, so it does not belong in the generic ctest suite. Move it to ci/run.sh as gg_run_test_fusion, gated on GG_BUILD_METAL like gg_run_test_llama_archs_tensor_split: it generates the dummy models with test-llama-archs -o and then validates the fusion counts against the committed baseline. test-fusion.cpp is still built (llama_build) but no longer registered as a ctest. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : align fusion baseline TSV columns Pad the TSV fields to fixed widths so the columns line up regardless of the variable arch and fusion-label lengths, and trim each field on parse so the padded file is still accepted. Regenerate the committed MTL.tsv baseline in the padded format (data unchanged, verified identical modulo padding). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : widen label column and align fusion TSV header Give the label column more room (28 chars) and fix the column header widths so they match the data rows (moe/mode/label), keeping the header aligned with the values. Regenerate the MTL.tsv baseline in the new format (data unchanged). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : switch fusion baseline from TSV to CSV Use comma-separated values like the rest of the project, keeping the padded, aligned columns. Split on ',' and trim on parse. Rename the committed baseline to MTL.csv (data unchanged, verified identical modulo padding/separator). Update the ci/run.sh check path accordingly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : rebase + update MTL stats * tests : avoid graph reallocations for some archs * metal : tidy fusion debugging context and op init - simplify the shared fusion debugging context comments - shorten the ggml_metal_fusion struct comment - align the ggml_metal_fuse struct fields and comments - move the fusion parameter of ggml_metal_op_init right after dev Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : dedup fusion baseline into any mode prefill and decode always produce the same per-graph fusion count, so store a single row per label with mode = "any" and the per-graph count instead of two rows. this halves the baseline size and keeps the check stable. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : move fusion model generation to a separate step the dummy models generated by test-llama-archs are reused by other tests, so generate them once in their own step instead of inside test_fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : bump nmse thold * models : fix plamo2 graph * tests : remove "skip" logic from test-fusion * tests : set qwen3tts dummy vocab to codec head size the dummy qwen3tts model used a vocab of 4096 while the codec head is 3072, so the graph padded the output with -inf which made the NMSE in test-fusion produce NaN. use the exact codec head size instead so the padding is not generated at all. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * tests : regen fusion baseline reflect the plamo2 graph fix, which changed its fusion pattern split (RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * ci : skip dummy model generation on OpenVINO test-llama-archs does not build on the OpenVINO platform, so do not try to generate the dummy models there. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 * cont : minor * tests : enable test-llama-archs on windows * cont : disable on windows + workaround * metal : naming nits * test-fusion : add instructions to update baseline * context : fix Kimi-K3 graph reserve * fusion : update MTL * cont : fix naming * metal : rework fusion info storage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : align fusion info API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * metal : use opaque fusion handle in ad-hoc API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * ci : move fusion test to dedicated workflow Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp * cont : run only on ggml changes * cont : simplify * fusion : remove multi-output stuff for now * ci : fix typo	2026-09-11 12:41:54 +03:00

Author

SHA1

Message

Date

Georgi Gerganov

5b59b83f4e

metal : add MoE and SSM_CONV fusion optimizations (#28948 )

* metal : add top-k MoE fusion

Adds a Metal fusion for SOFT_MAX + ARGSORT + GET_ROWS with optional
routing-weight normalization and scale, matching the top-k MoE fusion
available in the CUDA and Vulkan backends. The fused kernel writes the
selected expert ids and routing weights directly, eliding the separate
softmax, argsort, get-rows, sum-rows, clamp, div and scale kernels.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : add MoE weighted reduction fusion

Fuses MUL(experts, weights) plus the expert VIEW/ADD chain into one kernel
that computes the weighted sum directly. The graph_optimize hook keeps the
expert and weight buffers alive until the fused output so the allocator cannot
reuse them while the kernel is still reading them.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* tests : expose MoE weighted reduction in fusion baseline

Use 2 experts per token in the generated MoE test models so the Metal
MoE weighted reduction fusion (MUL + ADD) is exercised by test-fusion.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : fuse RMS_NORM + SCALE

Adds NORM/RMS_NORM + SCALE fusion to the Metal backend by reusing the
norm+mul kernel with a scalar scale flag. Adds test coverage for both
NORM+SCALE and RMS_NORM+SCALE and regenerates the fusion baseline.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use function constant for RMS_NORM + SCALE

Replaces the runtime use_scale karg with a Metal function constant. The
norm+mul kernel is compiled with FC_norm_use_scale=false for MUL fusion and
FC_norm_use_scale=true for SCALE fusion, so the fused kernel has no runtime
branch.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use function constant for top-k MoE with_norm

Replaces the runtime with_norm karg with a Metal function constant. The
top-k MoE kernel is compiled separately for the normalized and non-normalized
routing variants, removing the runtime branch.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : rename moe_weighted_reduction suffix to moe_reduce

Shortens the MoE weighted-reduction fusion identifiers, kernel, pipeline,
matcher, args struct, and test op name from moe_weighted_reduction to
moe_reduce.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : add MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion

Adds dense mat-vec activation fusion for sigmoid/silu and bias+softplus.
The mat-vec kernels apply the activation/bias epilogue via function
constants, avoiding the separate unary/add passes.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : revert MUL_MAT + UNARY and MUL_MAT + ADD + UNARY fusion

The mat-vec activation fusion regressed decode throughput on Qwen3.6-35B-A3B
by ~8% (tg32 81.5 vs 88.5 t/s). The regression is caused by loss of
concurrency: the standalone unary kernels previously overlapped with other
mat-vec work, while fusing the activation into the mat-vec kernel serializes
it on the critical path.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : add SSM_CONV + UNARY (silu) fusion

The SSM_CONV kernels apply silu directly via a function constant, eliding
the separate unary pass. Regenerates the fusion baseline.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : address fusion review comments

- Fix declaration/table alignment
- Rename top-k MoE kargs fields to val_clamp / val_scale
- Move moe-reduce alloc-deps handling into a general fusion helper
- Remove the public moe-reduce matcher API

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : fix unused parameter in top-k MoE fusion check

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : guard SSM_CONV fusion lookup behind use_fusion

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : track all fused outputs in graph reorder

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : keep top-k MoE logits alive until fused output

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : refactor alloc deps to pattern-driven approach

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : check fused kernel destination in concurrency tracking

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* meta : forward graph_optimize to underlying backends

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use vector for fusion table

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* meta : keep graph_optimize unimplemented

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* parallel : fix non-deterministic prompt selection

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* parallel : support dummy models and add global logits run hash

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : sync cross-device copies with destination completion event

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : avoid const_cast in fusion alloc deps

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : skip fusions with aliased sources

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : hide fusion pattern definition

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use vector fusion op sequences

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : drop redundant struct keywords

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : add alloc deps comment separator

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : generalize fusion output memory ranges

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : rename fusion out_offsets to outs

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : avoid dst vector in memory range check

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : optimize fusion matching and multi-output handling

- use pointer arithmetic for fusion info count lookup
- avoid heap allocations in top-k MoE and MoE reduce pattern matchers
- use fusion outs for multi-output subgraph checks

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* Revert "parallel : support dummy models and add global logits run hash"

This reverts commit 57c7caf941c1b43c270fd5009c9f175063522e96.

* fusion : update MTL.csv

* metal : unroll constant loops in top-k MoE kernel

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use function constants for top-k MoE n_expert and top_k

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : rename fusion kargs to scale and clamp

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use function constants for moe_reduce and ssm_conv

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* fusion : update MTL.csv

2026-09-19 13:14:44 +03:00

Georgi Gerganov

bbdd9f246e

tests : add fusion baseline README and broaden fusion CI triggers (#28893 )

* tests : add README for updating the per-backend fusion baselines

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* ci : trigger fusion on changes to test-llama-archs.cpp and src/models

the dummy models and their architectures drive the fusion baselines, so a
change to either can alter the per-fusion counters and should re-run the
fusion job.

Assisted-by: pi:llama.cpp/Qwen3.8-27B

* tests : merge the fusion build commands in the README

assisted-by: pi:llama.cpp/Qwen3.8-27B

* pi : require explicit permission before posting PR/issue comments

assisted-by: pi:llama.cpp/Qwen3.8-27B

2026-09-14 15:45:05 +03:00

Georgi Gerganov

a2878d30df

metal : single-source fusion table + fusion debug rework (#28164 )

* metal : rework fusion patterns into a single table

All fusable op patterns for the Metal backend are now declared once in a
fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer
(ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next,
compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext
for the structural checks, and differ only in the mode used for the pattern
check (STRUCTURAL at optimize time, since tensors are not allocated yet, and
FULL at compute time, including Metal buffer placement). This also protects the
snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph
optimization, which was previously unprotected.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : fix absolute output indices in fusion patterns

ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph
node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query
was passing a relative index (n_ops - 1). As a result the last node of every
pattern was not recognized as an output and was subjected to the elidable
use-count check, which failed for essentially all fusions. This silently
disabled the norm/MUL fusion and caused a ~5% token-generation regression.

Pass the absolute graph index of the last node instead.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : fuse gated_delta_net with cache cpy

Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net
kernel is followed by a cpy that scatters its recurrent state snapshots into
the KV cache, the kernel writes the snapshots straight into the cache buffer
and the trailing cpy is elided.

The gdn output has other consumers (the attn scores view), so unlike the
elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion
pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks,
making the pattern-specific check callback the sole validator. Packing
(ggml_metal_fuse_max) now matches on the same view-transparent node sequence
that the compute phase uses, so the gdn + cache cpy group is packed along with
any intermediate views and stays adjacent through the reorder.

The fused cpy is a view consumer of the gdn (it writes the cache directly),
so its mem-range is skipped in the encoder; the skip is restricted to CPY
nodes consuming the previous fused node through a view so other fusions are
unaffected.

Add test_gated_delta_net_cache_fusion and register 5 cases.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : drop is_view_consumer mem-range skip

The is_view_consumer skip was carried over from the upstream gated_delta_net
cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in
the concurrency tracker only ever adds a (conservative) memory barrier at the
fusion point. It can never remove a barrier, so it cannot introduce a race. The
worst case is one spurious barrier per gdn+cache-cpy fusion, which is within
run-to-run noise on Qwen3.5-0.8B Q8_0.

Dropping the check keeps the mem-range loop uniform for all fused groups.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : rename gated_delta_net fused state output args

Rename the fused cache-write kernel argument to match the rest of the kargs:
state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id
bid_state_out -> bid_out.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : rename raw fusion flag to unsafe

raw did not convey that the flag opts a fusion pattern out of the generic
elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks).
rename it to 'unsafe' to make explicit that the pattern's check callback is the
sole validator and must re-establish the safety guarantees itself.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : tidy fusion pattern checks and table

- const-correct ggml_metal_fuse_outputs buffer
- annotate unused check-callback parameters
- drop a redundant size_t cast
- align the ops/table initializers and add blank-line separation

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* metal : add generic fusion stats via ad-hoc proc-address API

Add a device-owned fusion context that lets a test tool count how many
times each fusion pattern fires and toggle fusion. It is exposed through
the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names
so the testing tool is backend-agnostic:

- ggml_backend_fusion_stats_init: start collecting fusion stats; when a
  context is created afterwards it registers the labels/counters and
  encodes single-threaded (n_cb == 0) so the counters are race-free
- ggml_backend_fusion_stats_reset / _get_stats / _set_enabled

The context lives on the metal device (not on the last backend context),
so counters accumulate across contexts and reads are always consistent.
The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE
and can be overridden by the test through set_enabled. Labels are
synthesized from the fuse table via ggml_metal_fuse_label (e.g.
"GATED_DELTA_NET+CPY").

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : add fusion count regression test with per-backend baseline

test-fusion runs every dummy model generated by test-llama-archs on a
single backend (single-threaded encoding, n_cb == 0) with fusion enabled
and disabled, and for each mode (prefill / decode) reports the per-fusion
counters and the NMSE between the fused and unfused logits, plus the NMSE
against a CPU reference.

A fusion pattern that silently stops matching (or fires when it should
not) is caught as a regression by comparing the counters against a
committed per-backend TSV baseline:

- --record writes the golden baseline, --check (default) validates it
- the unfused run doubles as a control: its counters must be all-zero
- NMSE is skipped when it is NaN or the arch is already broken on the
  device (e.g. plamo2 on Metal), so the count check is the hard gate
- baseline counts depend only on graph structure, not weights (verified
  stable across weight seeds)
- the fusion stats API is resolved through the ad-hoc get_proc_address
  mechanism with generic names; a backend that does not export it makes
  the test fail with an error

The committed MTL0.tsv baseline covers 110 dummy archs (298 rows).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : rename fusion api helpers to match stats_init signature

Align the test with the ad-hoc fusion stats API: fusion_stats_init no
longer takes an enable bool (stats are turned on by calling it), so the
proc-address wrappers and typedefs are renamed to the api_* convention.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : rename backend to device in fusion test CLI

The fusion test operates on a compute device (e.g. MTL0), not a backend,
so rename the --backend argument to --device and the backend_name
variable to device_name. Keep "backend" where it refers to the ggml
backend interface (the ad-hoc proc-address mechanism).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : add --model and --help to fusion test

--model FILE runs the fusion regression test over a single model file
instead of enumerating a --models DIR. --models and --model are mutually
exclusive. Also add a --help/-h option that prints the usage.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : use backend base name for fusion baseline output

The fusion test is invoked with a specific device name (e.g. MTL0), but
its output - the recorded baseline and the header it writes - should be
named after the backend base name (e.g. MTL, via ggml_backend_reg_name),
since the counters depend on the backend, not on the specific device
index. Rename the committed baseline to MTL.tsv.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : run fusion test from ci instead of ctest

The fusion test needs Metal and generates a lot of dummy models, so it
does not belong in the generic ctest suite. Move it to ci/run.sh as
gg_run_test_fusion, gated on GG_BUILD_METAL like
gg_run_test_llama_archs_tensor_split: it generates the dummy models with
test-llama-archs -o and then validates the fusion counts against the
committed baseline. test-fusion.cpp is still built (llama_build) but no
longer registered as a ctest.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : align fusion baseline TSV columns

Pad the TSV fields to fixed widths so the columns line up regardless of
the variable arch and fusion-label lengths, and trim each field on parse
so the padded file is still accepted. Regenerate the committed MTL.tsv
baseline in the padded format (data unchanged, verified identical modulo
padding).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : widen label column and align fusion TSV header

Give the label column more room (28 chars) and fix the column header
widths so they match the data rows (moe/mode/label), keeping the header
aligned with the values. Regenerate the MTL.tsv baseline in the new
format (data unchanged).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : switch fusion baseline from TSV to CSV

Use comma-separated values like the rest of the project, keeping the
padded, aligned columns. Split on ',' and trim on parse. Rename the
committed baseline to MTL.csv (data unchanged, verified identical modulo
padding/separator). Update the ci/run.sh check path accordingly.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* cont : rebase + update MTL stats

* tests : avoid graph reallocations for some archs

* metal : tidy fusion debugging context and op init

- simplify the shared fusion debugging context comments
- shorten the ggml_metal_fusion struct comment
- align the ggml_metal_fuse struct fields and comments
- move the fusion parameter of ggml_metal_op_init right after dev

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : dedup fusion baseline into any mode

prefill and decode always produce the same per-graph fusion count, so
store a single row per label with mode = "any" and the per-graph count
instead of two rows. this halves the baseline size and keeps the check
stable.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* ci : move fusion model generation to a separate step

the dummy models generated by test-llama-archs are reused by other tests,
so generate them once in their own step instead of inside test_fusion.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : bump nmse thold

* models : fix plamo2 graph

* tests : remove "skip" logic from test-fusion

* tests : set qwen3tts dummy vocab to codec head size

the dummy qwen3tts model used a vocab of 4096 while the codec head is
3072, so the graph padded the output with -inf which made the NMSE in
test-fusion produce NaN. use the exact codec head size instead so the
padding is not generated at all.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* tests : regen fusion baseline

reflect the plamo2 graph fix, which changed its fusion pattern split
(RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total).

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* ci : skip dummy model generation on OpenVINO

test-llama-archs does not build on the OpenVINO platform, so do not try
to generate the dummy models there.

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731

* cont : minor

* tests : enable test-llama-archs on windows

* cont : disable on windows + workaround

* metal : naming nits

* test-fusion : add instructions to update baseline

* context : fix Kimi-K3 graph reserve

* fusion : update MTL

* cont : fix naming

* metal : rework fusion info storage

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : align fusion info API

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* metal : use opaque fusion handle in ad-hoc API

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* ci : move fusion test to dedicated workflow

Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp

* cont : run only on ggml changes

* cont : simplify

* fusion : remove multi-output stuff for now

* ci : fix typo

2026-09-11 12:41:54 +03:00