* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
test | before | after
----------------+----------------+----------------
pp4096 | 460.99 ± 0.45 | 462.47 ± 0.34 (unchanged)
tg128 | 19.10 ± 0.04 | 19.56 ± 0.03 (+2.4%)
tg128 @ d30000 | 12.68 ± 0.04 | 12.69 ± 0.03 (unchanged)
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Add comment for case 288 in topk-moe.cu
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
* Address review comments
* hex-mm: fold mm quant tasks into the main matmul threads
* hex-mm: minor formatting fixes
* hex-mm: cleanup is_quant checks in dma dispatch
* hex-mm: fix dst-spad alignment
* hex-mm: move fp kernels in the hvx-mm-kernels header
* hex-mm: fuse with ADD
* hex-fa: factor out ukernels into separate headers and unify the rest
* hex-fa: move kernel-params compute into the host
* hex-fa: refactor vtcm alloc for consistency
* hex-fa: add support for FA_SELECT
* hex-fa: update tracing insrumentation to cover all functions
* hex-fa: update hvx fallback thresholds to recover t/g regressions
* hex-fa: update tracing instrumentation
* hex-fa: improved tracing with additional events
* hex-fa: optimize mask processing (fastdiv, etc)
* hex-fa: improve mask dma caching
* hmx-fa: change loop order to maximize mask cache hits
* hex-fa: remove over instrumentation
* hex-fa: breakdown QKV prep trace events
* hmx-fa: further mask proc optimizations
* hex-fa: mask broadcast is the common case, optimize for that
* hex-fa: use aligned loads where possible
* hex-fa: update loops to use uint32_t indices
* hmx-fa: fold vtcm init into q prep task
* hex-fa: update rest of the hmx funcs to use uint32_t
* hmx-fa: fold build_d into the main softmax loop
* hmx-fa: start kv dmas earlier
* hmx-fa: start mask dma a bit earlier
* hex-fa: precompute rows per task to avoid divs
* hmx-fa: specialize fa_o_store for f16 and f32
* hmx-fa: prelim support for Sinks
* hmx-fa: keep softmax accumulators in fp32
* hex-fa: add tanh_f16 and exp2_f16 and use that in FA
* hex-fa: use fp16 math in the hvx kernel
* hex-fa: avoid expensive float -> __fp16 cast for slopes and softcap
* hex-fa: replace most vec_exp_f32 with vec_exp2_f16
* hmx-fa: vectorize sinks update
* hex-fa: minor formatting
* hmx-fa: fold softcap loop into the tile load
* hmx-fa: use vectoralias to populate sinks
* hex-fa: remove redudant check
* hex-fa: fix vtcm size compute to use fp32 for accumulators
* hex-mm: fix trailing spaces
* hmx-fa: dont use -inf to init mask to avoid conversion overflows
* hex-fa: no need to explicitly guard -inf in the f16->f32 converter now
* hmx-fa: cleanup fa sinks handling
* hex-mm: fixed src2 stride handling when mm is fused with add
* hex-fa: make lto happy
* vulkan: roll bk loop in matmul for asahi linux
* vulkan: fix inline comment
* vulkan: revert BK-loop unroll change
* vulkan: edit spirv directly for asahi roll bk loop
* vulkan: remove trailing whitespace at the end of comments
* HIP: keep MMQ for gfx900 MoE and Q8_0, use hipBLAS for dense K-quants
Assisted-by: GitHub Copilot CLI
* HIP: tighten conditional block to be explicitly for gfx900
* HIP: Further simplified gfx900 conditional block
* removed unnecessary comment
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
* opencl: fix inifity for `-cl-finite-math-only`
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* [CUDA] Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy
Add a CUDA ggml_cpy fast path for same-type, same-shape strided copies that are just 2D pitched block copies.
When tensors are not fully contiguous but each row is contiguous, it now uses cudaMemcpy2DAsync instead of the slow element-wise scalar copy kernel.
This fixes the GDN recurrent snapshot update with -np 4, where rollback slots are separated by cache stride gaps.
* Add new tests that execute the new optimized strided copy path
* Return unsupported for strided copy in OpenVINO, as new tests are failing
* CUDA: Improve performance via less synchronizations between token (llama/17795)
* Adds CPU-to-CUDA copy capability to
ggml_backend_cuda_cpy_tensor_async()
* Adds function to relax sync requirements between input copies on
supported backends (CUDA for now)
* Exchanges synchronous copy with async copy function.
* Adds macro guards to allow compilation in non-CUDA builds
* Reworked backend detection in ggml-backend.cpp to avoid linking
conflicts
* Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues
* Minor cleanup
* Makes opt-in to relax use of explicit syncs more general. Backends like
vulkan which require a synchronization between HtoD copies and graph
execution could also adopt this change now.
* Reintroduces stricter check for CPU->CUDA backend async copy via
GGML_DEVICE_TYPE_CPU.
* Corrects initialization of ggml_backend_sync_mode in
ggml_backend_sched_split initialization
* Simplifies synchronizations to adhere to `saaasg` pattern.
* Apply suggestion from @ggerganov (src->buffer to buf_src)
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Apply suggestion from @ggerganov (src->buffer to buf_src) v2
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Apply suggestions from @johannesgaessler code review
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs.
* Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split ->
GPU split optimization
* Scheduler Hardening: Re-adding original additional synchronizations for
non-async backends
* Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of
precaution, but that no perf-impact is visible, and that it can be
revisited separately anytime.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* vulkan: add INTEL_PRE_XE2 arch enum and enable coopmat1 on Intel Xe-LPG Plus (1/3, Xe1-ARLH)
Co-authored-by: Xia, Jie <jie.xia@intel.com>
Co-authored-by: Liu, Russell <russell.liu@intel.com>
* Address comments of bf16 and trailing whitespace
* Rename INTEL_PRE_XE2 to INTEL_XE1 and remove driver workaround
* Add Windows driver check
---------
Co-authored-by: Xia, Jie <jie.xia@intel.com>
Co-authored-by: Liu, Russell <russell.liu@intel.com>
When transcribing with params.vad = true, whisper already computes the speech
segments and keeps them in the state. Expose them so callers can reuse those
boundaries (for example to align or clip subtitles to speech) instead of running
a second, separate VAD pass.
Times are on the original audio timeline in centiseconds; the count is 0 when VAD
was not used. test-vad-full.cpp checks the segments are ordered and non-empty.
* ci, examples : add package-lock.json to examples addon node
This commit commits the package-lock.json file for examples/addon.node.
This is needed to make the CI workflow work properly and after we
started pinning the action versions in the workflow, the CI started
failing because the package-lock.json was not committed. At least that
is what I think is happening.
* ci: use node 20.x and 22.x instead of 16.x and 18.x
* fix CMakeLists.txt target properties for addon.node
whisper_full_get_token_data().t0/t1 are in the VAD-processed timeline (silences
removed), so they don't line up with the original audio. Add
whisper_full_get_token_t0/t1 that map them back.
A token inside a speech segment is interpolated within it; a token that falls in a
removed inter-segment silence snaps to the nearest boundary, so it never lands in
the middle of a cut-out gap. Without VAD the raw times are returned unchanged.
* whisper : make voice_length() utf-8 aware for CJK
voice_length() weights each token by how long its text takes to say, which drives
how a segment's time is shared between its tokens. It looped over raw bytes, so
every CJK character (3 bytes) was counted ~3x and full-width punctuation never
matched, skewing token timestamps for Chinese/Japanese.
Decode one utf-8 code point at a time and give full-width ，。！？ etc. the same
weights as their ASCII counterparts. Pure-ASCII text is unaffected.
* whisper : one statement per line in voice_length()
---------
Co-authored-by: linxiaodong <calm.lin@wukongsch.com>
* fix/android-debug-inference-issue: Applied BUILD_TYPE=Release to NDK build of whisper.android example
* fix/android-debug-inference-issue: Applied BUILD_TYPE=Release to NDK build of whisper.android.java example
* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32
2D convolutions with kernel size 9 produced different results on SVE
enabled ARM devices. After debugging it turned out that ggml_vec_dot_f32
was using data from inactive lanes.
Use svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1.
* cont : clean-up
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Add failing test-case to test-backend-ops
Extracted from https://github.com/ggml-org/llama.cpp/issues/24072
* Minimize repro with help of AI
N = 8 * (65535 - 1) + 1 = 524273
* Port and adjust workaround from 0ba798341e
Fall-back should share code, also relax y-z constraint to be inclusive
* Add test-case + fallback also for y dim
* Fix x-guards which is 2^{31}-1, so inlusive of INT_MAX
* Fix overflow problems for transposed copy kernel
* Sycl tp stage1 (llama/1)
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16:
pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%)
tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%)
Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE):
pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%)
tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%)
Qwen3-4B Q4_K_M:
pp64 = 984.51 t/s, tg16 = 49.29 t/s
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
* documentation update to address PR feedback
* Bring over my device-to-device memcpy chagnes
* move the dev2dev_memcpy calls to the upstream 7-parameter variety
* Fix a typo and remove a trailing whitespace
* hex-mm: new weight layout and fusion updates
* hvx-mm: unroll the new tiled vec_dots to optimize hvx register util
* hex-mm: optimize dyn.quant format for q8_0 and q8_1 to reduce overhead in vec_dots.
* hvx-mm: parallel quantizer per block for large rows
* hvx-mm: simplify and futher optimize dyn.quant and vec_dots
* hvx-mm: keep intermediate per tile accumulators in fp16
* hmx-mm: optimize weight dequant by aligning the repacked tiles with the DMA
* hmx-mm: remove qweight scratch and just use vtcm_weight
* hmx-mm: remove all unused and obsolete code
* hmx-mm: the new tiled repack format is here to stay -- rename all x4x2 to _tiled
* hmx-mm: improve activation processing with dma prefetch
* hex-mm: fix hmx/hvx fallback logic and MUL_MAT_ID allocation (unbreaks OLMoE)
* hex-mm: align the weight tiles with dma just like we did in hmx-mm
* hex-mm: factor out common mm bits into htp/matmul-ops.h
* hex-mm: start moving mm kernel selection to the host
* hex-mm: move all of the matmul param compute into the host
* hmx-mm: restore pipelined mode
* hmx-mm: unroll the dequant functions to optimize register usage
* hmx-mm: further improve activation process
* hex-mm: use vtcm_seq_alloc for all vtcm allocations and define more common functions
* hex-mm: improve mm optimizer to acount for number of activation threads
* hex-mm: fix matmul-id kernel params selection (unbreaks OLMoE and LFM)
* hexagon: remove support for arch < v73 since HMX is now required for most use-cases
* hex-mm: cleanup naming for consistency
* hex-mm: make sure matmul fusion accounts for vtcm allocation
* hex-mm: minor cleanup for kernel_params definition
* hex-mm: replace hardcoded limits with proper checks for vtcm requirements
* hex-mm: add support for non-tiled mm as a fallback option and factor out hvx kernels into separate header
* hex-mm: remove unused functions
* hex-mm: add shorthand for MM_SELECT in run-tool script
* hvx-mm: factor out hvx/hmx microkernels and unify matmul entry and dispatch
* hex-mm: further cleanup matmul fallback path
* hex-mm: refactor matmul entry point and dispatch a bit further
* hexagon: update cmake build to enable hmx for everything
* hex-ops: optimize kernel_param updates and include summary in the logs
* hex-mm: add support for GGML_HEXAGON_MM_SELECT
* hex-mm: add hex-common header
* hex-mm: pass correct number of tasks to workpool
* hex-mm: add proper checks for no-work in dyn.quant tasks
* hex-mm: convert all quantizers into a macro
* hex-mm: fix hvx-flat fallback to pass all MUL_MAT tests
* hex-mm: vectorize q8_1 quantizer
* hex-mm: improve fused ffn mm stride handling
* hex-mm: consistent use of n_threads and pipeline in kernel_params
* hexagon: minor formatting
* hex-mm: update MUL_MAT_ID kernel_param handling to make sure host/npu are in sync
* hvx-mm: go back to accumulating in fp32 in tiled hvx kernels, more accurate and same perf
* hvx-mm: unroll the loops and remove masking that is not needed for tiled accums
* hmx-mm: optimize activation processing (slit loops, some unrolling, etc)
* hmx-mm: minor optimization for output processing
* hex-mm: consistent use of uint32_t and size_t in mm kernels
* hex-mm: remove legacy restrictions for rows to be multiple of 256
* hexagon: replace sprintf with snprintf
* hex-mm: relax hardcoded nrows checks and rely on VTCM size requirements
* hexagon: minor alignment fix
* hexagon: fix trailing spaces
* hex-mm: relax padding from 256 to 128 (leftovers)
* hex-mm: remove redundant checks for weight align to 128
we always use 2D dma for the weights and align them properly
* hmx-mm: MUL_MAT_ID better work distribution between hvx threads and hmx tracing
* hex-mm: specialize per-token mmid activation handling
* hex-profile: update python scripts to handle kernel-params section in the logging output
* hex-mm: move n_prefetch (aka dma_depth) into kernel params and remove unused fields
* hex-trace: use easier to parse format, simply and fix post-proc scripts
* hmx-mm: relax 32 row limit for output processing which helps utilization
* hmx-mm: use start-chunk idx for tracing info
* hmx-mm: parameterize activation dma pipeline
* hexagon: add support for simple graph caching to avoid recomputing kernel-params
* hex-mm: remove left-over repack functions
* hex-mm: tighten n_prefetch asserts
* hex-mm: remove duplicate round/align_up helper
* hexagon: cleanup common header used in host/npu
* hexagon: update early wakeup threshold
* hmx-mm: define cost constants and update solver to assume that repacked ne[1] is padded to 32
* hmx-mm: make precompute_matmul a bit more readable (split into smaller functions, etc)
* hex-mm: remove n_threads constraint
* hex-mm: minor formatting updates
* hex-mm: remove obsolete profiling logs
* hex-mm: restore hardcode gate to refuse lm-head to avoid repacking that tensor
* vulkan-shaders-gen: fail the build when a shader fails to compile
vulkan-shaders-gen did not detect shader-compile subprocess failures, so a
broken libggml-vulkan could be produced while the build reported success and
the breakage only surfaced at run time. execute_command() discarded the child
exit code (POSIX waitpid passed nullptr for status; the Windows branch never
called GetExitCodeProcess) and string_to_spv decided success only from whether
stderr was empty, so a non-zero exit with empty stderr, or a subprocess that
failed to launch, was treated as success.
Return the child exit code from execute_command() (WEXITSTATUS on POSIX,
GetExitCodeProcess on Windows), treat a non-zero exit or non-empty stderr or a
launch exception as a failure, and record it in an atomic flag. main() checks
the flag after process_shaders() and returns EXIT_FAILURE before writing the
output files, so the build stops instead of emitting a broken backend.
Fixes#24393
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* vulkan-shaders-gen: simplify compile_failed access and drop unreachable return
Address review feedback on #24450:
- Access the std::atomic<bool> compile_failed directly (= / implicit bool)
instead of .store()/.load(); the flag stays atomic because the worker
threads in process_shaders() set it concurrently.
- Remove the unreachable trailing return -1 in execute_command(): on POSIX the
child _exit()s after execvp and the parent returns (fork()<0 throws); on
Windows the block returns the exit code.
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
---------
Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
* vulkan: make SQR/SQRT/SIN/COS/CLAMP/LEAKY_RELU use unary.comp
* vulkan: make NORM support noncontig
* add noncontiguous row test cases for norm/l2_norm, handle this in the CPU backend and l2_norm.comp
* fix supports_op for cuda and webgpu
