upstream-archive/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-30 18:07:38 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,275 Commits 826 Branches 7,686 Tags
b11275
Commit Graph
11275 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Xiang Chen a6ea155d3d gguf : reject tensor size that wraps after padding (#26979) 
GGML_PAD(nbytes, alignment) wraps to 0 when nbytes is within
(alignment - 1) of SIZE_MAX, which silently bypassed the size
overflow guard in gguf_init_from_reader. Reject the tensor before
padding when nbytes + (alignment - 1) would overflow.

Adds a test-gguf handcrafted case (F32, ne = [4, 2^30-1, 2^30+1, 1])
whose ggml_nbytes = 2^64 - 16 lands in the wrap window. Fails on
master, passes with the guard.
b11275 		2026-09-29 22:46:20 +02:00
Adrien Gallouët d3954b9324 ggml : check row bounds in get_rows_back (#29575) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11274 		2026-09-29 22:44:41 +02:00
bosh 48de2a1bcb model : support classifier_pooling for rerankers (#29627) 
* model : support classifier_pooling for ModernBERT rerankers

Assisted-by: Claude Opus 5.5

* model : read classifier pooling type in load_hparams

Write classifier.pooling_type from _try_set_pooling_type whenever the
config has classifier_pooling, and read it in
llama_model_base::load_hparams. ModernBERT falls back to mean when it
is unspecified.

Assisted-by: Claude Opus 5.5

* conversion : only accept cls and mean for classifier_pooling

Assisted-by: Claude Opus 5.5

* model : rename classifier_pooling_type to pooling_type_cls

Assisted-by: Claude Opus 5.5
b11273 		2026-09-29 22:33:05 +02:00
Trivikram Reddy 7fee178464 hexagon: optimize concat op (#29673) 
* hex-concat: reduce pkts in gather/transpose hot loop

gather directly into dst buffer, use special instruction for gather sync

* hex-concat: use fastdiv

replace calls to sw divide with fastpath

* hex-concat: optimize DMA-HVX pipeline and add transpose helpers
b11272 		2026-09-29 12:52:10 -07:00
Pascal 6a2743f028 CUDA: bitonic argsort handles rows wider than one block (#28957) 
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread
per padded column, so any row above 1024 entries launched an invalid
block configuration. Each thread now owns several columns, every stage
of the network runs all owned columns before the barrier, and the block
is capped at 1024 threads. Shared memory becomes the only bound, which
supports_op checks against the device instead of a fixed 1024.

Rows up to 1024 run the same work as before. Bit-exact with the CUB
path on rows of 2048.
 2026-09-29 20:09:10 +02:00
thelittlefiremanandCarl Philipp Klemm 748d4225b9 ggml-cuda: HIP: optimize packed byte subtraction (__vsubss4 -> __vsub4) (#29478) 
* ggml-cuda: HIP: optimize non-saturating packed byte subtraction (`__vsubss4`)

* CI: ignore 1 spilled vgpr in fattn_vec

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
b11270 		2026-09-29 19:59:19 +02:00
Aaron Teo cee37ffea0 ci: add zdnn backend build but not test (#29541) 
* ci: add zdnn backend build but not test

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: attempt to run a ubuntu 26.04 container

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: set shell to bash

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: clean up comments

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ggml-zdnn: fix compiler errors

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* vendor: attempt to ignore warnings from vendored files

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
b11269 		2026-09-29 20:45:14 +03:00
lhez 6dbbac4429 opencl: fix get_tensor for q5_K adreno gemm_nonshuffle kernel (#29555) b11268 2026-09-29 20:44:42 +03:00
Ruben Ortlam 5c200e0c8d vulkan: Tune GDN kernel, fix Intel performance (#29476) 
* vulkan: tune GDN shader

* tune for Intel
b11267 		2026-09-29 20:44:20 +03:00
Matt Corallo 83dd71f869 vulkan : Load F32 A matrix 2 at a time when its 2-aligned (#29254) 
It turns out Intel doesn't particularly like loading F32s one at a
time and we already have the _2aliagned load logic in mul_mat_vec,
so here we use it.

While we do already check all the requirements to load elements 4
at a time across [B]F16 and F32, it turns out [B]F16 loading 4 at a
time is sometimes slower on very specific shapes on Intel BMG.
Loading 4 at a time is a bit faster on F32, but its not material
and I assume might be slower on other platforms.

Note that we also need to validate `a_offset` is 2-aligned in
`mul_mat_vec.comp`, which was missing in the original 2-way-load
patch.

Some selected speedups from `test-backend-ops perf` on a B60.

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   767.17 us/run - 117.44 MFLOP/run - 153.08 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    2556 runs -   529.81 us/run - 117.44 MFLOP/run - 221.66 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   727.13 us/run - 234.88 MFLOP/run - 323.03 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    2130 runs -   528.84 us/run - 234.88 MFLOP/run - 444.15 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   702.19 us/run - 352.32 MFLOP/run - 501.74 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1988 runs -   532.14 us/run - 352.32 MFLOP/run - 662.08 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1278 runs -   919.50 us/run - 469.76 MFLOP/run - 510.89 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1917 runs -   543.69 us/run - 469.76 MFLOP/run - 864.03 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1197 runs -   892.12 us/run - 587.20 MFLOP/run - 658.21 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1881 runs -   575.17 us/run - 587.20 MFLOP/run -   1.02 TFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1498 runs -   716.40 us/run - 939.52 MFLOP/run -   1.31 TFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1819 runs -   576.36 us/run - 939.52 MFLOP/run -   1.63 TFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                   134 runs -  7467.09 us/run -  60.13 GFLOP/run -   8.05 TFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                   134 runs -  7478.12 us/run -  60.13 GFLOP/run -   8.04 TFLOPS
b11266 		2026-09-29 20:39:36 +03:00
Ankit Khandelwal 94a0ae3e72 vulkan: MOE aware mat_mul_id tile selection (#29182) 
mut_mul_id selected its matmul tile with total token count.
For MoE dispatch grid the true N per workgroup is per-expert rows.
At pp128 on Sarvam 30B that is 6, not 128, so the picker took the l-tile for ~6 live rows.
Most workers in each group had nothing to do.
This wasted time. The slow part was 55% of the whole job.
b11265 		2026-09-29 20:38:59 +03:00
XZiarGeorgi GerganovXZiar
 		da89bb3ccc ggml : fix c++ odr by properly using GGML_COMMON_DECL_CPP (#29504) 
* fix c++ odr by properly using GGML_COMMON_DECL_CPP

* using actual field rather than macro

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: XZiar <xziar@xziar.xziar>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
b11264 		2026-09-29 20:31:31 +03:00
Toki Nasin a3f84faf49 vocab : keep </s> NORMAL in PLaMo-2 and PLaMo-3 (#29580) 
* vocab : keep </s> NORMAL in PLaMo-2 and PLaMo-3

The PLaMo-2 and PLaMo-3 vocabularies mark </s> as NORMAL. Current
EOG token heuristic matched it by text and added its attribute
to CONTROL.

Skip this heuristic for the PLAMO2 vocab type so </s> stays NORMAL
and is not treated as EOG.

* use <|plamo:eos|> for detection
b11263 		2026-09-29 20:30:40 +03:00
Adrien Gallouët 284153e069 ggml : accumulate f16 dot products in f32 on AVX512-FP16 (#29545) 
Supersedes #29530

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11262 		2026-09-29 20:29:12 +03:00
Georgi Gerganov b5cf8ce02a ggml : require input tensors to be GGML_OP_NONE (#29647) b11261 2026-09-29 20:23:33 +03:00
Aparna M P e904318a2d hexagon: add FP32 GELU_ERF and GEGLU_ERF support (#29631) 
* hexagon: add FP32 GELU_ERF and GEGLU_ERF support

* hex-erf: reduce register pressure in kernels
b11260 		2026-09-29 09:35:47 -07:00
Ethan Guo d280808f5d common : stop accepting draft tokens at EOG (#29638) 
* common : stop accepting draft tokens at EOG

* cont : remove the test
b11259 		2026-09-29 18:25:41 +02:00
Emanuil Rusev ba0ba54d93 server : remove the built-in UI's service worker when the UI is not served (#29565) 
With --path or --no-ui, /sw.js returned 404, and a 404 does not remove a service worker, so browsers kept showing the cached built-in UI. Serve a worker that unregisters itself, clears its caches and reloads open tabs. A sw.js in the --path folder is still served first.

Assisted-by: Claude Opus 5.5
b11258 		2026-09-29 17:48:17 +02:00
Adrien Gallouët 00af63567a common : use fs::path for config dir (#29649) 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11257 		2026-09-29 17:22:49 +02:00
Georgi Gerganov c85b92c69c tests : adjust server string regex to also match m2 utlra results (#29648) b11256 2026-09-29 15:32:35 +03:00
Adrien Gallouët 31385c9ceb common : add fs_write_atomic() (#29642) 
- Check for buffered write errors when closing downloaded files.
- Use UTF-8 paths when writing ETag files on Windows.
- Write in binary mode on Windows.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11255 		2026-09-29 13:46:53 +02:00
Georgi Gerganov 8019dc563b ggml : collect all input tensors into graph_inputs (#29634) 
graph_inputs was populated while splitting the graph, so it only
contained the inputs that are used as srcs of some node. With pipeline
parallelism (n_copies > 1), each graph input contributes n_copies leafs
to graph_copy, so switching between batches that consume different
inputs (e.g. token batches that do not use the embeddings input vs
image batches that do) changed the graph composition. This shifted the
input copies in graph_copy, making the backend ids comparison report
spurious changes and forcing the scheduler to re-reserve. The
re-reserve could then record smaller input sizes (e.g. out_ids with
n_outputs = 0) and abort later on a graph with an unchanged size via
GGML_SCHED_DEBUG_REALLOC.

Collect the inputs after the split instead, from all input leafs of the
graph, so that the graph composition depends only on which inputs
exist, not on which inputs are used.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
b11254 		2026-09-29 13:37:50 +03:00
Aaron Teo 86ea01d05e ggml-zdnn: fix 0-row tensor crash (#29636) 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-09-29 10:51:10 +02:00
R0CKSTARandyeahdongcn 18b74ff68e musa: build the docker image and CI container from the MUSA SDK images (#29624) 
* musa: build the docker image and CI container from the MUSA SDK images

Use registry.mthreads.com/mcconline/musa_sdk:5.2.0-{devel,runtime}-ubuntu22.04-s5000
instead of registry.mthreads.com/mcconline/inference/pytorch:2.9.1.post1-py3.10-musa5.2.0-mp31-devel-ubuntu22.04-amd64
for the MUSA docker image and the MUSA CI container, and let the runtime stage use the
runtime image instead of reusing the devel one, which drops the MUSA toolchain from the
published images.

* musa: install the MUSA headers and loader path the SDK images omit

musa_sdk:5.2.0-*-s5000 does not ship the cub and thrust headers that the MUSA
backend builds against, and its runtime image does not register
/usr/local/musa/lib with the dynamic loader.

Install both header packages in the build stage and in the MUSA CI container,
and write the loader path in the runtime stage.

* musa: install libmthreads-compute for the MUSA runtime library

The MUSA SDK images do not install libmthreads-compute, which provides
libmusa.so.1 in /usr/lib/x86_64-linux-gnu, so linking anything against the
MUSA backend fails.

* musa: install libmthreads-compute in the runtime stages

The MUSA runtime image does not install libmthreads-compute, so the published
images would have no libmusa.so.1 at run time.

---------

Co-authored-by: yeahdongcn <yeahdongcn@users.noreply.github.com>
 2026-09-29 10:42:20 +02:00
Adrien GallouëtandJohannes Gäßler c13e04e1dd ggml : speed up model loading (#29598) 
* ggml : speed up model loading

A crafted model could hang the server for a very long time, try with:

    llama-cli -hf angt/test-gguf-1Mkv -hff model.gguf

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Avoid empty keys

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2026-09-29 10:35:08 +02:00
Aaron Teo c8cda8b4fe ci: remove gpu-rocm keyed directory logs (#28940) 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-09-29 16:21:54 +08:00
Georgi Gerganov 6d78fb0727 llama : fix init in several tools/examples (#29632) b11249 2026-09-29 10:32:05 +03:00
bri-prism 18bbc46b46 metal: FWHT perf optimizations (#29602) 
Assisted-by: Claude Code
 2026-09-29 15:29:51 +08:00
Andrew Lee 0bc845d356 vulkan : reuse descriptor sets when bindings are constant (#29280) 
* vulkan : reuse descriptor sets when bindings are constant

* vulkan : bump buffer_destroy_count before destroying the buffer
b11247 		2026-09-29 08:54:55 +02:00
vaibhavdedhiaandAlde Rojas 139997d8e7 chat : fix Muse Glimmer ignoring response_format json_schema with --jinja (#29615) 
* chat : fix Muse Glimmer ignoring response_format json_schema with --jinja

Fixes #29613

* chat : accept json fences and clean up

* chat : fix choice parenthesis

---------

Co-authored-by: Alde Rojas <hello@alde.dev>
b11246 		2026-09-29 08:18:58 +02:00
Adrien Gallouët 76a5bc86d1 common : use fs::path for cache dirs (#29595) 
- Avoid useless string conversions on Windows.
- No need for BSD or emscripten special cases.

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11245 		2026-09-29 07:24:03 +02:00
Georgi Gerganov 46e17a6352 tests : skip pytest workers when PYTEST_WORKERS=1 (#29610) 
Assisted-by: pi:llama.cpp/Qwen3.8-27B
 2026-09-29 08:20:54 +03:00
Asahi-PrvandAsahi-Prv fc07d781e6 ci : update the oneAPI toolkit to 2026.1 (#29273) 
* ci : update the oneAPI toolkit to 2026.1

oneDNN is removed from Intel Deep Learning Essentials in 2026.0, so
staying on the deep-learning-essentials path would silently lose oneDNN
support when the toolkit version is updated. Switch both the Ubuntu and
Windows CI jobs to the new unified Intel oneAPI Toolkit installer,
which still includes oneDNN (until 2027.0) and keeps the component IDs
unchanged for the Windows install script.

Measured with the same code (b10899) built with oneAPI 2026.1 vs the
2025.3-based release build on Arc B570: prompt processing 1331 vs 434
t/s (3.1x), token generation 50.1 vs 45.3-48.0 t/s.

Assisted-by: GLM (z-ai/glm-5.3-flash)

* docs : update the SYCL backend build requirements for oneAPI 2026.1

With the 2026.0 release the Base toolkit and the HPC toolkit are
combined into the oneAPI Toolkit, and oneDNN is removed from the Deep
Learning Essentials package. Update the install instructions, the
verified release table and the news section accordingly.

Assisted-by: GLM (z-ai/glm-5.3-flash)

* ci : update the release workflow for oneAPI 2026.1 and Level Zero SDK 1.33.1

Align the release package build with the CI build update:
- oneAPI toolkit 2025.3.3 -> 2026.1 (the unified oneAPI Toolkit)
- Level Zero SDK 1.28.2 -> 1.33.1, and the Debian package names
  (level-zero/level-zero-devel -> libze1/libze-dev)
- The Windows DLL copy list for the 2026.1 runtime: sycl9.dll and the
  .6/.3 MKL library versions

Assisted-by: GLM (z-ai/glm-5.3-flash)

* ci : remove the removed .spv fallback files from the Windows DLL copy list

oneAPI 2026.1 no longer ships libsycl-fallback-bfloat16.spv and
libsycl-native-bfloat16.spv (the OpenCL fallback mechanism changed), so
the copy step failed with exit 1.

Assisted-by: GLM (z-ai/glm-5.3-flash)

* devops : update the oneAPI toolkit image in the Intel Dockerfile

Assisted-by: GLM (z-ai/glm-5.3-flash)

---------

Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
b11243 		2026-09-29 10:19:51 +08:00
Pascal 526c43b8f7 mtmd: fix GCC 15 stringop-overflow in decode_embd_batch (#29607) b11242 2026-09-29 01:21:37 +02:00
Trivikram Reddy 1c4729414d hex-scripts: show trace events smaller than 100nsec in perfetto (#29614) 2026-09-28 15:02:58 -07:00
Tim Wangtimothywang21Xuan Son Nguyen
 		680a036285 server : support typed content (vision/audio/video) input for /v1/embeddings endpoint (#29556) 
* server : support multimodal input for /v1/embeddings (Qwen3-VL-Embedding)

Accept the OpenAI-style wrapped content array format for multimodal
embedding requests. Each {"content": [...]} object is one input that
produces one embedding; text parts are concatenated and image_url parts
are decoded via handle_media then spliced with process_mtmd_prompt.

The legacy formats (plain string, token arrays, mixed arrays, and the
{prompt_string, multimodal_data} object) continue to work unchanged via
tokenize_input_prompts. Bare content arrays (the unwrapped shape) are
rejected with a migration message.

Also disables KV prefix reuse for stateless embedding/rerank tasks so
that repeated inputs do not incorrectly share cached KV across requests.

Assisted-by: Opencode Qwen3.8 27B

* clean up comments and docs

* refactor

* add tests

* support video and audio inp

---------

Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
b11240 		2026-09-28 21:40:38 +02:00
Linux User 66e665c427 vulkan: include functional header (#29597) 
Fixes the compile error: `no template named 'function' in namespace 'std'`
b11239 		2026-09-28 21:31:42 +02:00
Pascal 57b557cb95 models: pad on the left with ggml_pad_ext (#29567) 
* models: pad on the left with ggml_pad_ext

The Parakeet, LFM2-Audio, Granite Speech and Gemma 4 audio encoders
build a left padding as a right pad followed by a roll, and DFlash2
concatenates a zero filled block in front of the previous tokens.
ggml_pad_ext does both in one node now that every backend supports a
left padding. The Gemma 4 audio embeddings are bit identical.

* models: skip the DFlash2 taps that only read padding

A tap at or past block_size shifts every row out of the block, so its
term is zero. The loop runs min(kernel_size, block_size) taps.
b11238 		2026-09-28 20:56:15 +02:00
Ravi Panchumarthy 14ebbd5f2f ggml-openvino: mark unaligned batch-stride views unsupported (#29603) b11237 2026-09-28 21:54:01 +03:00
Xuan-Son Nguyen f1ea206218 batch: migrate speculative, mtmd and server to batch_ext (#29385) 
* adapt common

* add common_batch

* wip

* wip: spec

* cont

* common_speculative_process

* server_batch to use common_batch

* rm some stale calls

Assisted-by: Claude Fable 5.1

* migrate mtmd

* handle imrope, handle return val of add()/add_embd()

* add spec zeros vector

* add warning on zero fill path
b11236 		2026-09-28 19:52:45 +02:00
Adrien Gallouët 6c7a87f7e5 common : fix HF cache paths on Windows (#29475) 
Supersedes #29158

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
b11235 		2026-09-28 16:25:24 +02:00
jbooth f00a64c147 webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor (#29471) 
* Fix:  Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor

* Clang formatting
b11234 		2026-09-28 16:38:10 +03:00
Georgi Gerganov d77dd0806d tests : refactor test-recurrent-state-rollback (#29426) 
* tests : use llama_context_ptr in test-recurrent-state-rollback

Replace raw llama_context pointers with llama_context_ptr and drop the
manual llama_free calls and cleanup lambda.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : run test-recurrent-state-rollback over all dummy models

Add a --models DIR mode that mirrors test-save-load-state: iterate every
dummy model, report PASS/FAIL/SKIP in a table and fail only when a model
fails. Register a single ctest entry with ARGS --models instead of the four
per-model registrations.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* cont : fix typo

* metal : allow fusing 0-element nodes to keep graph packing shape-independent

The fusion packing in ggml_metal_fusion_max excluded 0-element tensors and
the topk_moe/moe_reduce checks rejected n_tokens == 0, so graphs decoding
batches with no outputs packed differently from the worst-case reserved
graph. The Metal optimizer then reordered the nodes differently and
ggml_gallocr_needs_realloc failed on the layout mismatch, forcing an
unexpected graph re-reserve (caught by GGML_SCHED_DEBUG_REALLOC).

Treat empty tensors like their non-empty counterparts: match them in the
pattern sequence and only reject genuinely malformed shapes. Fused kernels
dispatch zero threadgroups for empty graphs, which is a legal no-op.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : run test_multi_seq_split_replay as a separate test

test_multi_seq_split_replay was invoked at the end of test_rollback,
so its result was folded into the rollback status and it only ran when
the rollback part passed.

Give it its own test_status return, run both tests independently over
both cache fills via a shared run_tests helper, and report them as
separate rollback / split replay columns in the --models table with
per-test summaries. The exit code fails when either test fails.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : loosen the split replay nmse bound to 1e-4

test-generate-models seeds its weights from std::random_device, and some
generated lfm2 models drift up to ~1.7e-5 nmse on the split replay due to
rounding noise, tripping the previous 1e-5 bound. Raise the bound to 1e-4
so the random generations stop flaking.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : reuse run_tests_for_model in single-model mode

The single-model path duplicated the model init and the non-recurrent
check from run_tests_for_model; route it through the shared helper
instead. Model load failures now return FAIL rather than SKIP so that
--model with a broken file still exits non-zero, and the helper loads
with model_only like the --models loop does since the tests create
their own contexts.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
b11233 		2026-09-28 16:36:38 +03:00
SXX 6f767fe960 ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 (#29423) 
* ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86

* add AVX2 support for masked loading and storing in simd_gemm_ukernel_tail

* ggml-cpu: fix FA softcap handling for padded KV tiles
b11232 		2026-09-28 16:23:31 +03:00
uvos f916130d00 ci : ignore more vgpr spills in > 256 DQK fattn kernels (#29571) 2026-09-28 14:52:07 +02:00
François-Xavier Gsell 03a667aa30 vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain (#29520) 2026-09-28 14:20:14 +02:00
uvos c2a9e16068 HIP: fix template skip for DKQ > 256 mfma kernels (#29559) b11229 2026-09-28 13:47:29 +02:00
Pascal 4364bf7232 metal: support left and circular padding in GGML_OP_PAD (#29561) 
* metal: support left and circular padding in GGML_OP_PAD

Align Metal with CPU, CUDA and Vulkan: shift the source coordinates by
the left paddings, wrap them around with the same wrap_around when
circular, and read the source through nb00, which also fixes a right
padding of a permuted source. A test case covers it.

Drop the f32_4 kernel: its selection is disabled as slower, and it
fails two pad cases once enabled.

* metal: use a function constant for the circular pad variant

Address review from ggerganov: replace the bool template with FC_PAD,
as FC_upscale_aa does, so the pad kernel is compiled once and
specialized per pipeline.
b11228 		2026-09-28 12:26:50 +02:00
Sihan YuandGeorgi Gerganov ed7ac35e1e context : do not re-reserve the scheduler when toggling causal_attn (#28751) 
* context : do not re-reserve the scheduler when toggling causal_attn

`llama_context::set_causal_attn()` marks the scheduler to do a full re-reserve on every change of the flag. For vision inputs, this flag is flipped twice around each non-causal image chunk for Gemma models, resulting in two expensive `sched_reserve()` passes per image. This is especially slow for multi-image or video inputs.

The cost of a re-reserve scales with context and ubatch configurations, so larger settings pay more per image (see table below).

The re-reserve is unnecessary in this case because `causal_attn` only changes the values written to KQ mask, not tensor shapes or any other buffer sizes.

Note: `causal_attn` is a graph reuse key (`llm_graph_params` via `cparams`), so a new graph is built regardless of `sched_need_reserve`, so this doesn't change the graph rebuilding behaviour.

llama-server with gemma-4-26B-A4B Q4_0 + BF16 mmproj, 130-token images,
cache_prompt=false, prompt_ms median of 3 (before -> after):

| images | config | H200 before -> after | RTX 4090 before -> after |
|-|-|-|-|
| 1 | `-c 8192 -ub 512` | 134 -> 105 ms (1.27×) | 201 -> 119 ms (1.69×) |
| 24 | `-c 8192 -ub 512` | 2278 -> 1562 ms (1.46×) | 3559 -> 1748 ms (2.04×) |
| 24 | `-c 32768 -ub 2048` | 5379 -> 1584 ms (3.40×) | 13377 -> 1759 ms (7.61×) |

Generated output remains identical before and after.

* qwen4exp : make the indexer bias shape independent of causal_attn

The block/cell bias path was selected on cparams.causal_attn, so the
causal and non-causal graphs differed in tensor shapes and ops. With the
re-reserve removed (previous commit), a runtime flip resulted in
reallocating the compute buffers, which would fail under
GGML_SCHED_NO_REALLOC.

This commit selects the block path from the mask shape only, independent
of causal_attn. causal_attn is instead passed to set_input_qsa.
causal_attn is fixed per graph as it's part of the reuse key. Causal
values are unchanged. Non-causal values now follow the reference rule,
where every visible block competes on score and only unpooled cells are
always selected.

* context : state the causal_attn shape rule in the comment

* cont : add TODOs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
b11227 		2026-09-28 11:58:51 +03:00
Sarah Wu 0c6a6a7ce5 Enables Windows ARM64 build with MSVC cl.exe (#28362) 
* can reproduce the issue vlad sees

* fix fma issue

* drop volatile

* fix volatile runtime task

* add arm flag if needed

* fix hsum compile error

* fix syntax in quants

* strengthen sve probing

* make the syntax fixes one liners

* remove debug code

* formatting

* remove macro for float

* drive down gcc instruction count

* support armec

* fix CI comments

address CI comments

fix cross compile issue

remove warning

fix style and fix fma probing

fix style

* add documentation

* update documentation
b11226 		2026-09-28 10:07:27 +02:00