Compare commits

..
19 Commits
Author SHA1 Message Date
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
ac830a9cbb ggml : add backend scheduler tests (#27258)
* Adds initial scheduler stress tests

* Adds new sleep op to simplify scheduler testing

* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321

* Adds experimental vulkan GGML_OP_SLEEP

* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.

* Cleanup: Removes CUDA specific comments and nvtx-markers

* Fix: forgot to add sleep specific test case

* Fix oversights identified by CI

* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior

* Fix: Early exit in CPU-only settings (cause for graviton CI fails)

* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds

* Exclude ACCEL devices from new tests due to incompatibility

* Fixes vulkan test failure in llvmpipe

* Adds asserts, improves comments and some checks

* Fixes nitpicks in white-space and variable naming

* Adds comments that certain vk drivers could time OP_SLEEP incorrectly

* Drops limitiation to CPU/GPU in test-sleep

* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)

* Implements feedback for OP_SLEEP to work in place (no copy)

* Apply non-code suggestions  from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail

* Make clang-tidy happy

* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)

* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name

* Drops unnecessary shader_clock.comp test and guards

* Replaces sched_ prefixes with helper_ prefix

* Moves test suite to new RAII types for better memory management

* Adds sleep option to test_chain_all_backends

* Adds sleep option to first-run arm of the y-shaped test

* Fix switchcase order for legibility in stress_test_dag

* Simplfies stress_test_linked_list via double for-loop

* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)

* Fix RPC Proto patch due to GGML_OPS bump

* Adds ggml-cpp.h treatment to test-sleep.cpp

* Adds TODO pointers for subsequent additional tests.

* Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

* Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Completes callsite change of gg-suggestion

* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 16:40:13 +02:00
Piotr Wilkin (ilintar) 8af68d486a llama: fixes for Inkling 2 attention mask (#32429)
* llama : give Inkling2 its own relative-position attention input

The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.

Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.

Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* inkling2 : make the expert weight scale and final logit softcap optional

Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : build kv_rows masks for the hybrid iswa memory

llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : make the relative-position input a hybrid iswa input

Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : compose the hybrid iswa rel input, format inkling2

Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.

Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:36:30 +01:00
Alexey Kopytko f8caa5ea30 metal : fix the second argument of NSMakeRange (#32602) 2026-11-19 13:49:22 +02:00
SS1618 4db8b7638f webgpu: add POOL_2D support (#32250)
* Shader, pipeline, and launcher

* Update docs and add newline to wgsl

* Formatting fixes
2026-11-19 13:35:10 +02:00
Georgi Gerganov 01592dc58b llama : remove redundant fields from llm_graph_params (#32596)
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-19 13:31:50 +02:00
Georgi Gerganov 8bf2132c3f readme : fix release badge (#32597) 2026-11-19 13:20:47 +02:00
Adrien Gallouët 5812201a81 ci : fix build-opencl.yml (#32594)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 11:24:10 +01:00
PascalandGeorgi Gerganov 41d8297fe4 metal : CONV_2D tensor API for M5, split-K for all chips (#32542)
* metal: add a tensor API conv_2d kernel and split long reductions

The conv_2d matmul kernel runs on simdgroup matrices only, and gives
each threadgroup the whole reduction of its output tile. On M5 this
leaves the tensor cores unused, and convolutions with few output
positions and a long reduction (IC * KW * KH in the thousands) run a
handful of threadgroups on an otherwise idle GPU.

Add kernel_conv_2d_mm_tensor for devices with the tensor API: like
kernel_mul_mm, it gathers the patches of 64 output positions into
threadgroup memory as the left operand and reads the kernel rows
from device memory as the right operand. Devices without the tensor
API keep the simdgroup kernel.

Both kernels can now split the reduction until there are enough
threadgroups: each split writes its partial sums to a buffer placed
after the output, as flash attention does, and a small kernel adds
them in a fixed order, so the result stays deterministic. CONV_2D is
marked as an op whose allocation may exceed its output size.

test-backend-ops -o CONV_2D passes, with and without
GGML_METAL_TENSOR_DISABLE.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     19008 -> 5235
  [16,16,128,8] k [3,3,128,512]         333 -> 124
  [1536,1536,32,1] k [3,3,32,64]      12086 -> 6512
  [33,1,1280,1] k [3,1,1280,1280] s2    659 -> 33
  [2054,1,768,1] k [7,1,768,768]       3161 -> 699

* metal: walk the conv_2d patches incrementally, size splits by output

The patch gather of both conv_2d matmul kernels spent most of its
time on address math: 64-bit multiplies, divisions and four bound
checks per value. Walk the reduction index with 32-bit byte offsets
inside the image, updated by additions, with one unsigned compare per
bound; the host checks that an image fits. The tensor kernel no
longer gathers the reduction indices past the end of its split.

The number of splits now targets about 1M partial sums, with at least
3 reduction steps per split, instead of counting the tiles of one
kernel.

CONV_2D, F16 kernel, us/run on an M5 Max:
  [19,19,256,16] k [4,4,256,4096]     4998 -> 4512
  [1536,1536,32,1] k [3,3,32,64]      6721 -> 5354
  [4096,1,1024,1] k [7,1,1024,1024]   2701 -> 2236
  [2054,1,768,1] k [7,1,768,768]      1050 -> 871
  with GGML_METAL_TENSOR_DISABLE:
  [4096,1,1024,1] k [7,1,1024,1024]   9047 -> 8283

* nit: share the CONV_2D simdgroup count between host and kernel

kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per
threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel
constant and res.nsg, so the dispatch and the kernel cannot drift apart.

* metal: fall back to the generic conv_2d kernel for images past 4 GiB

The matmul conv_2d kernels walk the patches with 32-bit byte offsets
inside an image. Instead of asserting on nb13, compute the largest
offset actually read from ne/nb, which also covers strided views of
the input, and route images that do not fit to the generic kernel.

* cont : comments

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 11:48:44 +02:00
saiph 51a77c4f24 qwen3vl : set t_layer_inp for EAGLE-3 (#32589)
Assisted-by: Grok Bot
2026-11-19 10:13:02 +01:00
Bartowski 76f186df71 ggml : IQ2 creation speedup through caching (#31910)
* IQ2 quant creation speedup from caching

* Rename variables for clarity

* Explicit casts

* Initialize all values, add comment to tolerance
2026-11-19 10:04:58 +02:00
Adrien Gallouët 95b09b62be llama : remove PATH_MAX (#32557)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 08:57:57 +01:00
Adrien Gallouët b5dbca1e44 ci : use LLVM 22 (#32565)
* ci : fix build-vulkan.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* ci : fix behavior-check.yml

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 08:54:04 +01:00
Aleksander Grygier 80437488a4 ui: cut Llama Chat's first-load JavaScript by 63%, load diagrams, math and PDFs on demand (#32555)
* ui: import Lucide icons by path in modules that run in Node

Assisted-by: Claude (Opus 5.5 Medium)

* ui: register highlight.js languages once, from lowlight's modules

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load the pdf.js worker and nerdamer as files of their own

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw Mermaid diagrams with Pele

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin dependency versions and use one Svelte across the UI packages

Assisted-by: Claude (Opus 5.5 Medium)

* ui: build Llama Chat as split chunks

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load KaTeX and pdf.js on first use

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-19 08:37:35 +01:00
Adrien Gallouët 39039aa27f llama : fall back to buffered I/O on EIO (#32545)
* llama : fall back to buffered I/O on EIO

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Preserve bytes already read, resumes at the current file offset

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-19 07:50:33 +01:00
Birk Skyum 7a4117eef9 vulkan : enable MMQ with mixed signedness integer dots (#31045)
* vulkan : enable MMQ with mixed signedness integer dots

* vulkan: specialize MMQ shaders for mixed-signedness dot products

* vulkan: track integer dot product support with signed, mixed and unsigned flags

* vulkan: move signed dot check to the MMVQ call sites
2026-11-19 08:15:09 +02:00
Jeff Bolz 573aa89de6 vulkan: Implement stream-k MUL_MAT (#28528)
* vulkan: Implement stream-k MUL_MAT

Stream-k tries to split chunks of 256 K elements evenly across all SMs.
Workgroups that finish a tile write to dst, others write to scratch, and
there's a resolve shader similar to split-k.

The shader code is implemented for scalar+cm1+cm2, but only currently enabled
for cm2. The heuristic for enabling on coopmat2 is pretty simple due to the
large tile size (one workgroup per SM), but would be more complex for the tile
sizes being used for cm1, and it doesn't make sense for me to tune that on
NVIDIA devices.

Assisted-by: Codex

* address review feedback
2026-11-19 08:14:14 +02:00
Daniil Sterkhov ecafc67145 vulkan: use floating-point zero for storage-only FP16 in DIAG/TRI (#32379)
* vulkan: Change integer zero to floating-point zero for storage-only FP16 in DIAG/TRI.

Assisted-by: OpenAI Codex

* delete narrow test
2026-11-19 08:13:06 +02:00
Aman Gupta d4f5c26049 CUDA: use 64-bit dst offsets in mul_mat_q (#32374)
The MMQ kernels computed the dst tile offset (jt*J*stride_col_dst) in
32-bit int. For a dst with more than 2^31 elements, column tiles that
start past INT_MAX/nrows wrap around and write out of bounds.

This happens with an all-logits output head, e.g. llama-perplexity
-c 32768 -b 32768 -ub 16384 on a 248320-vocab model: the q8_0 output
matmul has a 248320 x 16384 dst (4.07e9 elements) and aborts with an
illegal memory access.

Widen the offsets in mul_mat_q and the stream-k fixup to int64_t, and
add a (disabled by default, ~9 GB) test-backend-ops case.

Assisted-by: Claude Opus 5.5
2026-11-19 08:11:44 +02:00
97 changed files with 4224 additions and 1974 deletions
+5
View File
@@ -140,6 +140,11 @@ jobs:
source .venv/Scripts/activate
pip install -r requirements/requirements-convert_hf_to_gguf.txt
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
shell: cmd
+5
View File
@@ -61,6 +61,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Install OpenCL Headers and Libs
id: install_opencl
run: |
+5
View File
@@ -213,6 +213,11 @@ jobs:
run: |
choco install ninja
- name: Install LLVM
run: |
choco install llvm --version=22.1.8
echo "C:\Program Files\LLVM\bin" >> $env:GITHUB_PATH
- name: Build
id: cmake_build
run: |
+1 -1
View File
@@ -7,7 +7,7 @@
<b>LLM inference in C/C++</b>
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?filter=v*&color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v1)
[![Release](https://img.shields.io/github/v/release/ggml-org/llama.cpp?color=brightgreen)](https://github.com/ggml-org/llama.cpp/releases?q=tag:v1)
[![Nightly](https://img.shields.io/github/v/release/ggml-org/llama.cpp?label=nightly&filter=b*&color=orange)](https://github.com/ggml-org/llama.cpp/releases?q=b)
[![Server](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/server.yml?label=Server)](https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml)
[![Docker](https://img.shields.io/github/actions/workflow/status/ggml-org/llama.cpp/docker.yml?label=Docker)](https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml)
+1 -1
View File
@@ -90,7 +90,7 @@ Legend:
| PAD | ❌ | 🟡 | ✅ | 🟡 | ❌ | ✅ | 🟡 | ✅ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| PAD_REFLECT_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ |
| POOL_1D | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| REGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ❌ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| RELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| REPEAT | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | 🟡 | ❌ | ❌ |
+128 -128
View File
@@ -1155,134 +1155,134 @@
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i32,ne=[1,8,1,3],nr23=[1,1],r=2,v=0","support","0","no","WebGPU"
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i64,ne=[1,8,1,3],nr23=[1,1],r=2,v=1","support","0","no","WebGPU"
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i32,ne=[1,8,1,3],nr23=[1,1],r=2,v=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[10,3,2,1],k0=1,s0=1,p0=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[11,1,3,2],k0=1,s0=1,p0=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[128,2,1,3],k0=1,s0=1,p0=0","support","0","no","WebGPU"
Can't render this file because it is too large.
+2 -2
View File
@@ -8,10 +8,10 @@ extern "C" {
#define RPC_PROTO_MAJOR_VERSION 10
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_PATCH_VERSION 0
#define RPC_PROTO_PATCH_VERSION 1
#ifdef __cplusplus
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
#endif
#define GGML_RPC_MAX_SERVERS 16
+8
View File
@@ -429,6 +429,8 @@ extern "C" {
GGML_OP_GLU,
GGML_OP_SLEEP,
GGML_OP_COUNT,
};
@@ -2599,6 +2601,12 @@ extern "C" {
int n_tasks,
void * userdata);
// Busy-wait operator for scheduling and backend testing.
GGML_API struct ggml_tensor * ggml_sleep(
struct ggml_context * ctx,
struct ggml_tensor * a, // wait after this tensor has executed, return value is simply a full view of this tensor
int32_t us); // amount of time to wait in microseconds
// loss function
GGML_API struct ggml_tensor * ggml_cross_entropy_loss(
+1
View File
@@ -2584,6 +2584,7 @@ bool ggml_op_alloc_size_may_expand(enum ggml_op op) {
case GGML_OP_ARGSORT:
case GGML_OP_TOP_K:
case GGML_OP_ARGMAX:
case GGML_OP_CONV_2D:
return true;
default:
return false;
+5
View File
@@ -2034,6 +2034,10 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
{
ggml_compute_forward_fill(params, tensor);
} break;
case GGML_OP_SLEEP:
{
ggml_compute_forward_sleep(params, tensor);
} break;
case GGML_OP_FLASH_ATTN_EXT:
{
ggml_compute_forward_flash_attn_ext(params, tensor);
@@ -2306,6 +2310,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
case GGML_OP_REPEAT:
case GGML_OP_REPEAT_BACK:
case GGML_OP_LEAKY_RELU:
case GGML_OP_SLEEP:
{
n_tasks = 1;
} break;
+13
View File
@@ -2238,6 +2238,19 @@ void ggml_compute_forward_fill(const ggml_compute_params * params, ggml_tensor *
}
}
// ggml_compute_forward_sleep
void ggml_compute_forward_sleep(const ggml_compute_params * params, ggml_tensor * dst) {
if (params->ith != 0) {
return;
}
// same clock as ggml_time_us() callers; on macOS steady_clock is CLOCK_MONOTONIC_RAW, which can run faster
const int64_t t_end = ggml_time_us() + ggml_get_op_params_i32(dst, 0);
while (ggml_time_us() < t_end) {}
}
// ggml_compute_tri
template <typename T>
+1
View File
@@ -84,6 +84,7 @@ void ggml_compute_forward_top_k(const struct ggml_compute_params * params, struc
void ggml_compute_forward_leaky_relu(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_tri(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_fill(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_sleep(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_flash_attn_ext(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_flash_attn_back(
const struct ggml_compute_params * params,
@@ -1,17 +1,17 @@
#include "batch-get.cuh"
#include "batch-copy.cuh"
#include "common.cuh"
#include "ggml-cpp.h"
#include <climits>
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
struct ggml_cuda_batch_get_descriptor {
struct ggml_cuda_batch_copy_descriptor {
const unsigned char * src;
unsigned char * dst;
size_t size;
};
static __global__ void ggml_cuda_batch_get_kernel(const ggml_cuda_batch_get_descriptor * copies, size_t count) {
static __global__ void ggml_cuda_batch_copy_kernel(const ggml_cuda_batch_copy_descriptor * copies, size_t count) {
// Blocks along y cooperate on each descriptor.
const size_t index = blockIdx.x;
if (index >= count) {
@@ -50,11 +50,11 @@ static __global__ void ggml_cuda_batch_get_kernel(const ggml_cuda_batch_get_desc
}
}
struct ggml_cuda_batch_get_mapping {
struct ggml_cuda_batch_copy_mapping {
void * host = nullptr;
unsigned char * device = nullptr;
ggml_cuda_batch_get_mapping(void * host, size_t size) {
ggml_cuda_batch_copy_mapping(void * host, size_t size) {
const cudaError_t err = cudaHostRegister(host, size, cudaHostRegisterMapped | cudaHostRegisterPortable);
if (err != cudaSuccess) {
(void) cudaGetLastError();
@@ -70,10 +70,10 @@ struct ggml_cuda_batch_get_mapping {
}
}
ggml_cuda_batch_get_mapping(const ggml_cuda_batch_get_mapping &) = delete;
ggml_cuda_batch_get_mapping & operator=(const ggml_cuda_batch_get_mapping &) = delete;
ggml_cuda_batch_copy_mapping(const ggml_cuda_batch_copy_mapping &) = delete;
ggml_cuda_batch_copy_mapping & operator=(const ggml_cuda_batch_copy_mapping &) = delete;
~ggml_cuda_batch_get_mapping() {
~ggml_cuda_batch_copy_mapping() {
if (host) {
// Release only the registration acquired by this object.
CUDA_CHECK(cudaHostUnregister(host));
@@ -82,11 +82,12 @@ struct ggml_cuda_batch_get_mapping {
};
#endif
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
static bool ggml_cuda_copy_tensor_n(int device, cudaMemcpyKind direction,
const ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
#if defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
GGML_UNUSED(device);
GGML_UNUSED(direction);
GGML_UNUSED(tensors);
GGML_UNUSED(data);
GGML_UNUSED(offsets);
@@ -94,14 +95,17 @@ bool ggml_cuda_get_tensor_n(int device,
GGML_UNUSED(n_tensors);
return false;
#else
const bool to_host = direction == cudaMemcpyDeviceToHost;
// Provisional dispatch values based on experiments.
constexpr size_t min_batch_reads = 128;
constexpr size_t max_gather_read_bytes = 128 * 1024;
// Collect non-empty CPU destination ranges.
constexpr size_t max_kernel_read_bytes = 128 * 1024;
// Collect destination ranges; repeated sources are allowed.
std::vector<std::pair<uintptr_t, uintptr_t>> ranges;
ranges.reserve(n_tensors);
size_t payload_bytes = 0;
// Number of non-empty reads whose size is at most max_gather_read_bytes.
uintptr_t begin = UINTPTR_MAX;
uintptr_t end = 0;
// Number of non-empty reads whose size is at most max_kernel_read_bytes.
size_t n_reads_small = 0;
for (size_t i = 0; i < n_tensors; ++i) {
if (sizes[i] == 0) {
@@ -111,15 +115,19 @@ bool ggml_cuda_get_tensor_n(int device,
GGML_ASSERT(address <= UINTPTR_MAX - sizes[i]);
GGML_ASSERT(payload_bytes <= SIZE_MAX - sizes[i]);
payload_bytes += sizes[i];
n_reads_small += sizes[i] <= max_gather_read_bytes;
ranges.emplace_back(address, address + sizes[i]);
n_reads_small += sizes[i] <= max_kernel_read_bytes;
begin = std::min(begin, address);
end = std::max(end, address + sizes[i]);
const uintptr_t destination = to_host ? address : (uintptr_t) tensors[i]->data + offsets[i];
GGML_ASSERT(destination <= UINTPTR_MAX - sizes[i]);
ranges.emplace_back(destination, destination + sizes[i]);
}
if (ranges.empty()) {
return true;
}
// Total number of non-empty reads.
const size_t n_reads = ranges.size();
if (n_reads > SIZE_MAX / sizeof(ggml_cuda_batch_get_descriptor) || n_reads > INT_MAX) {
if (n_reads > SIZE_MAX / sizeof(ggml_cuda_batch_copy_descriptor) || n_reads > INT_MAX) {
return false;
}
if (n_reads < 2 || (n_reads < min_batch_reads && n_reads_small == n_reads)) {
@@ -127,72 +135,85 @@ bool ggml_cuda_get_tensor_n(int device,
}
std::sort(ranges.begin(), ranges.end());
for (size_t i = 1; i < ranges.size(); ++i) {
// Preserve the ordered copy path when CPU destinations overlap.
// Preserve the ordered copy path when destinations overlap.
if (ranges[i].first < ranges[i - 1].second) {
return false;
}
}
// Find the CPU span to register, including gaps between destinations.
const uintptr_t begin = ranges.front().first;
const uintptr_t end = ranges.back().second;
if (end - begin - payload_bytes > payload_bytes) {
// Limit gaps in the CPU span without underflow when sources overlap.
if (end - begin > payload_bytes && end - begin - payload_bytes > payload_bytes) {
return false;
}
// Register and map the existing CPU destination so the kernel can write to it directly.
ggml_cuda_batch_get_mapping mapping((void *) begin, end - begin);
// Register and map the CPU span for direct kernel access.
ggml_cuda_batch_copy_mapping mapping((void *) begin, end - begin);
if (!mapping.host) {
return false;
}
// Reads selected for gather; zero means all reads use DMA.
size_t n_reads_gather = mapping.device && n_reads_small >= min_batch_reads ? n_reads_small : 0;
// Reads selected for the kernel; zero means all reads use DMA.
size_t n_reads_kernel = mapping.device && n_reads_small >= min_batch_reads ? n_reads_small : 0;
// Keep descriptor storage alive until the final stream synchronization.
constexpr size_t max_batch_descriptors = 128 * 1024;
const size_t bytes = std::min(n_reads_gather, max_batch_descriptors) * sizeof(ggml_cuda_batch_get_descriptor);
ggml_backend_buffer_ptr descriptors(n_reads_gather ? ggml_backend_buft_alloc_buffer(ggml_backend_cuda_buffer_type(device), bytes) : nullptr);
const size_t bytes = std::min(n_reads_kernel, max_batch_descriptors) * sizeof(ggml_cuda_batch_copy_descriptor);
ggml_backend_buffer_ptr descriptors(n_reads_kernel ? ggml_backend_buft_alloc_buffer(ggml_backend_cuda_buffer_type(device), bytes) : nullptr);
if (!descriptors) {
n_reads_gather = 0;
n_reads_kernel = 0;
}
const auto host_type = ggml_backend_cuda_host_buffer_type();
ggml_backend_buffer_ptr host_descriptors(n_reads_gather ? ggml_backend_buft_alloc_buffer(host_type, bytes) : nullptr);
ggml_backend_buffer_ptr host_descriptors(n_reads_kernel ? ggml_backend_buft_alloc_buffer(host_type, bytes) : nullptr);
if (!host_descriptors || ggml_backend_buffer_get_type(host_descriptors.get()) != host_type) {
n_reads_gather = 0;
n_reads_kernel = 0;
}
if (n_reads_gather) {
auto * copies = static_cast<ggml_cuda_batch_get_descriptor *>(ggml_backend_buffer_get_base(host_descriptors.get()));
auto * device_copies = static_cast<ggml_cuda_batch_get_descriptor *>(ggml_backend_buffer_get_base(descriptors.get()));
if (n_reads_kernel) {
auto * copies = static_cast<ggml_cuda_batch_copy_descriptor *>(ggml_backend_buffer_get_base(host_descriptors.get()));
auto * device_copies = static_cast<ggml_cuda_batch_copy_descriptor *>(ggml_backend_buffer_get_base(descriptors.get()));
const int threads = 256;
const size_t nsm = ggml_cuda_info().devices[device].nsm;
size_t i = 0;
for (size_t first = 0; first < n_reads_gather; first += max_batch_descriptors) {
const size_t n_batch = std::min(n_reads_gather - first, max_batch_descriptors);
for (size_t first = 0; first < n_reads_kernel; first += max_batch_descriptors) {
const size_t n_batch = std::min(n_reads_kernel - first, max_batch_descriptors);
size_t count = 0;
for (; i < n_tensors && count < n_batch; ++i) {
if (sizes[i] == 0 || sizes[i] > max_gather_read_bytes) {
if (sizes[i] == 0 || sizes[i] > max_kernel_read_bytes) {
continue;
}
copies[count++] = {(const unsigned char *) tensors[i]->data + offsets[i],
mapping.device + ((uintptr_t) data[i] - begin), sizes[i]};
auto * host = mapping.device + ((uintptr_t) data[i] - begin);
auto * tensor = (unsigned char *) tensors[i]->data + offsets[i];
copies[count++] = {to_host ? tensor : host, to_host ? host : tensor, sizes[i]};
}
const size_t blocks_per_copy = std::min<size_t>(UINT16_MAX, std::max<size_t>(1, (nsm + n_batch - 1) / n_batch));
const dim3 blocks((unsigned int) n_batch, (unsigned int) blocks_per_copy);
// Stream ordering keeps each kernel ahead of the next descriptor overwrite.
CUDA_CHECK(cudaMemcpyAsync(device_copies, copies, n_batch * sizeof(copies[0]), cudaMemcpyHostToDevice, cudaStreamPerThread));
ggml_cuda_batch_get_kernel<<<blocks, threads, 0, cudaStreamPerThread>>>(device_copies, n_batch);
ggml_cuda_batch_copy_kernel<<<blocks, threads, 0, cudaStreamPerThread>>>(device_copies, n_batch);
CUDA_CHECK(cudaGetLastError());
if (first + n_batch < n_reads_gather) {
if (first + n_batch < n_reads_kernel) {
// Wait before refilling the pinned host descriptor buffer.
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
}
}
// Keep the destination pinned for DMA too; overlapping destinations already declined above.
// Keep the CPU span pinned for DMA too; overlapping destinations already declined above.
for (size_t i = 0; i < n_tensors; ++i) {
if (sizes[i] && (!n_reads_gather || sizes[i] > max_gather_read_bytes)) {
CUDA_CHECK(cudaMemcpyAsync(data[i], (const char *) tensors[i]->data + offsets[i], sizes[i], cudaMemcpyDeviceToHost, cudaStreamPerThread));
if (sizes[i] && (!n_reads_kernel || sizes[i] > max_kernel_read_bytes)) {
void * host = const_cast<void *>(data[i]);
void * tensor = (char *) tensors[i]->data + offsets[i];
CUDA_CHECK(cudaMemcpyAsync(to_host ? host : tensor, to_host ? tensor : host, sizes[i], direction, cudaStreamPerThread));
}
}
// Wait for GPU writes before releasing the CPU mapping and descriptor buffers.
// Wait for transfers before releasing the CPU mapping and descriptor buffers.
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
return true;
#endif
}
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
return ggml_cuda_copy_tensor_n(device, cudaMemcpyDeviceToHost, tensors, data, offsets, sizes, n_tensors);
}
bool ggml_cuda_set_tensor_n(int device,
ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
return ggml_cuda_copy_tensor_n(device, cudaMemcpyHostToDevice, tensors, data, offsets, sizes, n_tensors);
}
@@ -5,3 +5,7 @@
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors);
bool ggml_cuda_set_tensor_n(int device,
ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors);
+16 -1
View File
@@ -10,7 +10,7 @@
#include "ggml-cuda/arange.cuh"
#include "ggml-cuda/argmax.cuh"
#include "ggml-cuda/argsort.cuh"
#include "ggml-cuda/batch-get.cuh"
#include "ggml-cuda/batch-copy.cuh"
#include "ggml-cuda/binbcast.cuh"
#include "ggml-cuda/clamp.cuh"
#include "ggml-cuda/col2im-1d.cuh"
@@ -48,6 +48,7 @@
#include "ggml-cuda/rope.cuh"
#include "ggml-cuda/roll.cuh"
#include "ggml-cuda/scale.cuh"
#include "ggml-cuda/sleep.cuh"
#include "ggml-cuda/snake.cuh"
#include "ggml-cuda/softcap.cuh"
#include "ggml-cuda/softmax.cuh"
@@ -1116,6 +1117,11 @@ static void ggml_backend_cuda_buffer_set_tensor_n(ggml_backend_buffer_t buffer,
ggml_cuda_set_device(ctx->device);
if (!ggml_cuda_pinned_memory_disabled() &&
ggml_cuda_set_tensor_n(ctx->device, tensors, data, offsets, sizes, n_tensors)) {
return;
}
for (size_t i = 0; i < n_tensors; i++) {
if (sizes[i] == 0) {
continue;
@@ -3145,6 +3151,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
case GGML_OP_FILL:
ggml_cuda_op_fill(ctx, dst);
break;
case GGML_OP_SLEEP:
ggml_cuda_op_sleep(ctx, dst);
break;
case GGML_OP_LIGHTNING_INDEXER:
ggml_cuda_lightning_indexer(ctx, dst);
break;
@@ -6809,6 +6818,12 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
return ggml_cuda_lightning_indexer_supported(dev_ctx->device, op);
case GGML_OP_TOP_K_MASK:
return op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_F32;
case GGML_OP_SLEEP:
#if defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
return false;
#else
return true;
#endif // defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
default:
return false;
+4 -4
View File
@@ -1094,7 +1094,7 @@ static __global__ void mul_mat_q(
int col_high = min(col_low + J, ncols_dst);
int offset_y = wt*stride_sample_y + zt*stride_channel_y;
int offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
int64_t offset_dst = (int64_t) wt*stride_sample_dst + (int64_t) zt*stride_channel_dst + (int64_t) jt*J*stride_col_dst;
if (ids_dst) {
const ggml_cuda_mm_id_tile_spec tsj = tile_spec[jt];
@@ -1176,7 +1176,7 @@ static __global__ void mul_mat_q(
int col_high = min(col_low + J, ncols_dst);
int offset_y = wt*stride_sample_y + zt*stride_channel_y;
int offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
int64_t offset_dst = (int64_t) wt*stride_sample_dst + (int64_t) zt*stride_channel_dst + (int64_t) jt*J*stride_col_dst;
if (ids_dst) {
const ggml_cuda_mm_id_tile_spec tsj = tile_spec[jt];
@@ -1257,7 +1257,7 @@ static __global__ void mul_mat_q(
int col_high = min(col_low + J, ncols_dst);
int offset_y = wt*stride_sample_y + zt*stride_channel_y;
int offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
int64_t offset_dst = (int64_t) wt*stride_sample_dst + (int64_t) zt*stride_channel_dst + (int64_t) jt*J*stride_col_dst;
if (ids_dst) {
const ggml_cuda_mm_id_tile_spec tsj = tile_spec[jt];
@@ -1403,7 +1403,7 @@ static __global__ void mul_mat_q_stream_k_fixup(
const int it = tmp2.x;
if (!ids_dst) {
const int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst + it*I;
const int64_t offset_dst = (int64_t) wt*stride_sample_dst + (int64_t) zt*stride_channel_dst + (int64_t) jt*J*stride_col_dst + it*I;
dst += offset_dst;
const int i_max = nrows_x - it*I - 1;
+32
View File
@@ -0,0 +1,32 @@
#include "sleep.cuh"
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
// %globaltimer is a nanosecond wall clock, unlike clock64() it is unaffected by the SM clock and by frequency scaling
static __device__ __forceinline__ uint64_t globaltimer_ns() {
uint64_t t;
asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(t));
return t;
}
// a single thread is enough, the next kernel on the same stream cannot start before this one retires
static __global__ void sleep_ns(const uint64_t ns) {
const uint64_t t0 = globaltimer_ns();
while (globaltimer_ns() - t0 < ns) {}
}
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
sleep_ns<<<1, 1, 0, ctx.stream()>>>(1000*(uint64_t) ggml_get_op_params_i32(dst, 0));
CUDA_CHECK(cudaGetLastError());
}
#else
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
GGML_UNUSED(ctx);
GGML_UNUSED(dst);
GGML_ABORT("GGML_OP_SLEEP requires the %%globaltimer register, which is only available on CUDA");
}
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
+3
View File
@@ -0,0 +1,3 @@
#include "common.cuh"
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
+35 -1
View File
@@ -2551,6 +2551,26 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm(ggml_
char base[256];
char name[256];
const bool has_tensor = ggml_metal_device_get_props(ggml_metal_library_get_device(lib))->has_tensor;
if (has_tensor) {
// nr0 output positions per threadgroup, nr1 output channels, same tiles as kernel_mul_mm
snprintf(base, 256, "kernel_conv_2d_mm_tensor_%s_%s", ggml_type_name(op->src[0]->type), ggml_type_name(op->src[1]->type));
snprintf(name, 256, "%s", base);
ggml_metal_pipeline_with_params res = ggml_metal_library_get_pipeline(lib, name);
if (!res.pipeline) {
res = ggml_metal_library_compile_pipeline(lib, base, name, nullptr);
}
res.nr0 = SZ_SIMDGROUP * N_MM_BLOCK_Y * N_MM_SIMD_GROUP_Y;
res.nr1 = SZ_SIMDGROUP * N_MM_BLOCK_X * N_MM_SIMD_GROUP_X;
res.nsg = N_MM_SIMD_GROUP_X * N_MM_SIMD_GROUP_Y;
res.smem = res.nr0 * N_MM_NK_TOTAL * sizeof(ggml_fp16_t);
return res;
}
const int nr0 = op->src[0]->ne[3] <= 32 ? 32 : 64;
const int nr1 = 2048/nr0;
@@ -2564,13 +2584,27 @@ ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm(ggml_
res.nr0 = nr0;
res.nr1 = nr1;
res.nsg = 4;
res.nsg = N_SG_CONV_2D;
res.smem = nr0*nr1*sizeof(float);
return res;
}
ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm_reduce(ggml_metal_library_t lib, const ggml_tensor * op) {
assert(op->op == GGML_OP_CONV_2D);
GGML_UNUSED(op);
const char * name = "kernel_conv_2d_mm_reduce";
ggml_metal_pipeline_with_params res = ggml_metal_library_get_pipeline(lib, name);
if (!res.pipeline) {
res = ggml_metal_library_compile_pipeline(lib, name, name, nullptr);
}
return res;
}
ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_dw(ggml_metal_library_t lib, const ggml_tensor * op, bool tiled) {
assert(op->op == GGML_OP_CONV_2D_DW);
+1
View File
@@ -170,6 +170,7 @@ struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_col2im_1d
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_snake (ggml_metal_library_t lib, enum ggml_type type);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_mm_reduce (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_2d_dw (ggml_metal_library_t lib, const struct ggml_tensor * op, bool tiled);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_conv_3d (ggml_metal_library_t lib, const struct ggml_tensor * op);
struct ggml_metal_pipeline_with_params ggml_metal_library_get_pipeline_upscale (ggml_metal_library_t lib, const struct ggml_tensor * op);
+1 -1
View File
@@ -2375,7 +2375,7 @@ void ggml_metal_buffer_memset_tensor(ggml_metal_buffer_t buf, struct ggml_tensor
id<MTLBlitCommandEncoder> encoder = [cmd_buf blitCommandEncoder];
[encoder fillBuffer:bid_dst.metal
range:NSMakeRange(bid_dst.offs, bid_dst.offs + size)
range:NSMakeRange(bid_dst.offs, size)
value:value];
[encoder endEncoding];
+8
View File
@@ -17,6 +17,8 @@
#define N_ARGMAX_NPART_MAX 256
#define N_ARGMAX_CHUNK 4096
#define N_SG_CONV_2D 4
// kernel parameters for mat-vec threadgroups
//
// N_R0: number of src0 rows to process per simdgroup
@@ -752,8 +754,14 @@ typedef struct {
int32_t p1;
int32_t d0;
int32_t d1;
int32_t nk; // reduction length per split
} ggml_metal_kargs_conv_2d;
typedef struct {
int32_t ne; // output elements
int32_t nsplit;
} ggml_metal_kargs_conv_2d_reduce;
typedef struct {
uint64_t nb00; // kernel strides
uint64_t nb01;
+69 -3
View File
@@ -4959,6 +4959,33 @@ int ggml_metal_op_im2col(ggml_metal_op_t ctx, int idx) {
return 1;
}
// number of reduction splits of the conv_2d matmul kernels
// A small output with a long reduction leaves the GPU idle, so the reduction is split until there are about 1M partial sums.
static int ggml_metal_op_conv_2d_nsplit(const ggml_tensor * op, int32_t * nk) {
const int64_t K = op->src[0]->ne[0]*op->src[0]->ne[1]*op->src[0]->ne[2];
const int64_t nsteps = (K + 31)/32;
const int64_t ne = ggml_nelements(op);
// at least 3 reduction steps of 32 per split, at most 32 splits
const int64_t nsplit = std::max<int64_t>(1, std::min<int64_t>({ (1024*1024 + ne - 1)/ne, nsteps/3, 32 }));
const int64_t steps = (nsteps + nsplit - 1)/nsplit;
if (nk) {
*nk = (int32_t) (steps*32);
}
return (int) ((nsteps + steps - 1)/steps);
}
size_t ggml_metal_op_conv_2d_extra_tmp(const ggml_tensor * op) {
assert(op->op == GGML_OP_CONV_2D);
const int nsplit = ggml_metal_op_conv_2d_nsplit(op, nullptr);
// partial sums of the reduction splits
return nsplit > 1 ? nsplit*ggml_nelements(op)*sizeof(float) : 0;
}
int ggml_metal_op_conv_2d(ggml_metal_op_t ctx, int idx) {
ggml_tensor * op = ctx->node(idx);
@@ -5012,11 +5039,17 @@ int ggml_metal_op_conv_2d(ggml_metal_op_t ctx, int idx) {
/*.p1 =*/ p1,
/*.d0 =*/ d0,
/*.d1 =*/ d1,
/*.nk =*/ 0,
};
const ggml_metal_device_props * props_dev = ggml_metal_device_get_props(ctx->dev);
if (props_dev->has_simdgroup_mm) {
// the matmul kernels walk the patches with 32-bit byte offsets inside an image
// larger images, possibly a strided view, take the generic kernel
const uint64_t max_off = (ne12 - 1)*nb12 + (ne11 - 1)*nb11 + (ne10 - 1)*nb10 + sizeof(float);
const bool fits_32 = max_off <= UINT32_MAX;
if (props_dev->has_simdgroup_mm && fits_32) {
auto pipeline = ggml_metal_library_get_pipeline_conv_2d_mm(lib, op);
const int nr0 = pipeline.nr0;
@@ -5026,15 +5059,48 @@ int ggml_metal_op_conv_2d(ggml_metal_op_t ctx, int idx) {
const int64_t NP = ne3*ne1*ne0;
const int64_t OC = ne03;
const int nsplit = ggml_metal_op_conv_2d_nsplit(op, &args.nk);
ggml_metal_buffer_id bid_dst = ggml_metal_get_buffer_id(op);
ggml_metal_buffer_id bid_tmp = bid_dst;
bid_tmp.offs += ggml_nbytes(op);
ggml_metal_encoder_set_pipeline(enc, pipeline);
ggml_metal_encoder_set_bytes (enc, &args, sizeof(args), 0);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op->src[0]), 1);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op->src[1]), 2);
ggml_metal_encoder_set_buffer (enc, ggml_metal_get_buffer_id(op), 3);
ggml_metal_encoder_set_buffer (enc, nsplit > 1 ? bid_tmp : bid_dst, 3);
ggml_metal_encoder_set_threadgroup_memory_size(enc, pipeline.smem, 0);
ggml_metal_encoder_dispatch_threadgroups(enc, (NP + nr1 - 1)/nr1, (OC + nr0 - 1)/nr0, 1, 32, nsg, 1);
if (props_dev->has_tensor) {
// per image: nr0 positions times nr1 channels per threadgroup
const int64_t OHW = ne1*ne0;
ggml_metal_encoder_dispatch_threadgroups(enc, (OC + nr1 - 1)/nr1, (OHW + nr0 - 1)/nr0, ne3*nsplit, 32, nsg, 1);
} else {
ggml_metal_encoder_dispatch_threadgroups(enc, (NP + nr1 - 1)/nr1, (OC + nr0 - 1)/nr0, nsplit, 32, nsg, 1);
}
if (nsplit > 1) {
GGML_ASSERT(ggml_is_contiguous(op));
// sync the 2 kernels
ggml_metal_op_concurrency_reset(ctx);
ggml_metal_kargs_conv_2d_reduce args0 = {
/*.ne =*/ (int32_t) ggml_nelements(op),
/*.nsplit =*/ nsplit,
};
auto pipeline0 = ggml_metal_library_get_pipeline_conv_2d_mm_reduce(lib, op);
ggml_metal_encoder_set_pipeline(enc, pipeline0);
ggml_metal_encoder_set_bytes (enc, &args0, sizeof(args0), 0);
ggml_metal_encoder_set_buffer (enc, bid_tmp, 1);
ggml_metal_encoder_set_buffer (enc, bid_dst, 2);
ggml_metal_encoder_dispatch_threadgroups(enc, (args0.ne + 255)/256, 1, 1, 256, 1, 1);
}
return 1;
}
+2
View File
@@ -49,6 +49,8 @@ size_t ggml_metal_op_flash_attn_ext_extra_rows(const struct ggml_tensor * op);
size_t ggml_metal_op_argmax_extra(const struct ggml_tensor * op);
size_t ggml_metal_op_conv_2d_extra_tmp(const struct ggml_tensor * op);
int ggml_metal_op_concat (ggml_metal_op_t ctx, int idx);
int ggml_metal_op_repeat (ggml_metal_op_t ctx, int idx);
int ggml_metal_op_acc (ggml_metal_op_t ctx, int idx);
+4
View File
@@ -311,6 +311,10 @@ static size_t ggml_backend_metal_buffer_type_get_alloc_size(ggml_backend_buffer_
{
res = 2*sizeof(int32_t)*ggml_nelements(tensor->src[0]);
} break;
case GGML_OP_CONV_2D:
{
res += ggml_metal_op_conv_2d_extra_tmp(tensor);
} break;
case GGML_OP_ARGMAX:
{
res += ggml_metal_op_argmax_extra(tensor);
+188 -13
View File
@@ -285,7 +285,7 @@ kernel void kernel_conv_2d_mm(
ushort tiitg[[thread_index_in_threadgroup]],
ushort tiisg[[thread_index_in_simdgroup]],
ushort sgitg[[simdgroup_index_in_threadgroup]]) {
constexpr short NSG = 4;
constexpr short NSG = N_SG_CONV_2D;
constexpr short NTH = 32*NSG;
constexpr short NK = 32;
constexpr short NKB = NK/8;
@@ -300,6 +300,10 @@ kernel void kernel_conv_2d_mm(
const int K = args.IC*args.KH*args.KW;
const int NP = args.N*args.OH*args.OW;
// this threadgroup reduces [k_begin, k_end), the splits are summed by kernel_conv_2d_mm_reduce
const int k_begin = tgpig.z*args.nk;
const int k_end = min(K, k_begin + args.nk);
const int r0 = tgpig.y*NR0;
const int r1 = tgpig.x*NR1;
@@ -308,8 +312,14 @@ kernel void kernel_conv_2d_mm(
const short lb_n = 8*sgitg + tiisg%8;
const short lb_k = 8*(tiisg/8);
// the patches are walked with 32-bit byte offsets inside the image, the host checks that an image fits
const uint nb10 = args.nb10;
const uint nb11 = args.nb11;
const uint nb12 = args.nb12;
int ix0[NPL];
int iy0[NPL];
uint off0[NPL];
bool ip_ok[NPL];
device const char * srcb[NPL];
@@ -324,6 +334,8 @@ kernel void kernel_conv_2d_mm(
ix0[t] = iow*args.s0 - args.p0;
iy0[t] = ioh*args.s1 - args.p1;
off0[t] = iy0[t]*nb11 + ix0[t]*nb10;
ip_ok[t] = ip < NP;
srcb[t] = src + in*args.nb13;
@@ -337,7 +349,7 @@ kernel void kernel_conv_2d_mm(
mc[i] = make_filled_simdgroup_matrix<float, 8>(0.f);
}
for (int k0 = 0; k0 < K; k0 += NK) {
for (int k0 = k_begin; k0 < k_end; k0 += NK) {
threadgroup_barrier(mem_flags::mem_threadgroup);
{
@@ -349,7 +361,7 @@ kernel void kernel_conv_2d_mm(
const int oc = r0 + row;
half v = 0;
if (kk < K && oc < args.OC) {
if (kk < k_end && oc < args.OC) {
v = *((device const TK *)(weights + oc*args.nb03) + kk);
}
@@ -364,7 +376,11 @@ kernel void kernel_conv_2d_mm(
int kw = kk % args.KW;
int kh = (kk / args.KW) % args.KH;
int ic = kk / (args.KW*args.KH);
int dx = kw*args.d0;
int dy = kh*args.d1;
// byte offset of the reduction index kk, relative to the patch origin
uint offk = (kk / (args.KW*args.KH))*nb12 + dy*nb11 + dx*nb10;
FOR_UNROLL (short i = 0; i < 8; i++) {
const short k = lb_k + i;
@@ -372,12 +388,12 @@ kernel void kernel_conv_2d_mm(
FOR_UNROLL (short t = 0; t < NPL; t++) {
const short n = 32*t + lb_n;
const int ix = ix0[t] + kw*args.d0;
const int iy = iy0[t] + kh*args.d1;
const int ix = ix0[t] + dx;
const int iy = iy0[t] + dy;
half v = 0;
if (ip_ok[t] && kk < K && ix >= 0 && ix < args.IW && iy >= 0 && iy < args.IH) {
v = *(device const float *)(srcb[t] + ic*args.nb12 + iy*args.nb11 + ix*args.nb10);
if (ip_ok[t] && kk < k_end && (uint) ix < (uint) args.IW && (uint) iy < (uint) args.IH) {
v = *(device const float *)(srcb[t] + (off0[t] + offk));
}
const short ib = NKB*(n/8) + k/8;
@@ -386,11 +402,17 @@ kernel void kernel_conv_2d_mm(
}
kk++;
dx += args.d0;
offk += args.d0*nb10;
if (++kw == args.KW) {
kw = 0;
kw = 0;
dx = 0;
dy += args.d1;
offk += args.d1*nb11 - args.KW*args.d0*nb10;
if (++kh == args.KH) {
kh = 0;
ic++;
kh = 0;
dy = 0;
offk += nb12 - args.KH*args.d1*nb11;
}
}
}
@@ -398,7 +420,7 @@ kernel void kernel_conv_2d_mm(
threadgroup_barrier(mem_flags::mem_threadgroup);
const short nkb = min((K - k0 + 7)/8, (int) NKB);
const short nkb = min((k_end - k0 + 7)/8, (int) NKB);
for (short ik = 0; ik < nkb; ik++) {
FOR_UNROLL (short i = 0; i < 4; i++) {
@@ -436,7 +458,8 @@ kernel void kernel_conv_2d_mm(
const int oh = (p / args.OW) % args.OH;
const int n = p / (args.OW*args.OH);
device char * dstp = dst + n*args.nb3 + oh*args.nb1 + ow*args.nb0;
// with several splits, dst is the partial sum buffer, one contiguous output per split
device char * dstp = dst + (uint64_t) tgpig.z*args.N*args.nb3 + n*args.nb3 + oh*args.nb1 + ow*args.nb0;
for (short j = 0; j < NR0*NR1/NTH; j++) {
const short row = tiitg/NR1 + (NTH/NR1)*j;
@@ -448,6 +471,158 @@ kernel void kernel_conv_2d_mm(
}
}
#ifdef GGML_METAL_HAS_TENSOR
// conv_2d as a matmul on the tensor cores, like kernel_mul_mm
// The patches of NRA output positions are gathered into threadgroup memory, the kernel rows are read from device memory.
// One threadgroup computes NRA positions of one image times NRB output channels, over the reduction range of its split.
template <typename TK>
kernel void kernel_conv_2d_mm_tensor(
constant ggml_metal_kargs_conv_2d & args,
device const char * weights,
device const char * src,
device char * dst,
threadgroup char * shmem [[threadgroup(0)]],
uint3 tgpig[[threadgroup_position_in_grid]],
uint3 tgpg[[threadgroups_per_grid]],
ushort tiitg[[thread_index_in_threadgroup]]) {
constexpr int NRB = SZ_SIMDGROUP * N_MM_BLOCK_X * N_MM_SIMD_GROUP_X; // output channels
constexpr int NRA = SZ_SIMDGROUP * N_MM_BLOCK_Y * N_MM_SIMD_GROUP_Y; // output positions
constexpr int NK = N_MM_NK_TOTAL;
constexpr int NTH = N_SIMDWIDTH * N_MM_SIMD_GROUP_X * N_MM_SIMD_GROUP_Y;
// each thread gathers NK/(NTH/NRA) consecutive reduction indices of one position
constexpr int KPT = NK/(NTH/NRA);
const int K = args.IC*args.KH*args.KW;
const int OHW = args.OH*args.OW;
const int nsplit = tgpg.z/args.N;
const int img = tgpig.z/nsplit;
const int split = tgpig.z%nsplit;
const int k_begin = split*args.nk;
const int k_end = min(K, k_begin + args.nk);
const int ra = tgpig.y*NRA;
const int rb = tgpig.x*NRB;
threadgroup half * sa = (threadgroup half *) shmem;
// the position this thread gathers
const int row = tiitg%NRA;
const int kc = (tiitg/NRA)*KPT;
const int p = ra + row;
const bool pok = p < OHW;
const int ix0 = (p%args.OW)*args.s0 - args.p0;
const int iy0 = (p/args.OW)*args.s1 - args.p1;
device const char * srcb = src + img*args.nb13;
// the patch is walked with 32-bit byte offsets inside the image, the host checks that an image fits
const uint nb10 = args.nb10;
const uint nb11 = args.nb11;
const uint nb12 = args.nb12;
const uint off0 = iy0*nb11 + ix0*nb10;
device TK * ptrB = (device TK *) weights;
const int strideB = args.nb03/sizeof(TK);
mpp::tensor_ops::matmul2d<
mpp::tensor_ops::matmul2d_descriptor(
NRB, NRA, static_cast<int>(dynamic_extent), false, true, true,
mpp::tensor_ops::matmul2d_descriptor::mode::multiply_accumulate),
execution_simdgroups<N_MM_SIMD_GROUP_X * N_MM_SIMD_GROUP_Y>> mm;
auto tB0 = tensor(ptrB, dextents<int32_t, 2>(K, args.OC), array<int, 2>({1, strideB}));
auto tA0 = tensor(sa, dextents<int32_t, 2>(NK, NRA));
auto cT = mm.get_destination_cooperative_tensor<decltype(tB0), decltype(tA0), float>();
for (int k0 = k_begin; k0 < k_end; k0 += NK) {
// the reduction indices past k_end are beyond the extent given to the matmul, they are not gathered
if (k0 + kc < k_end) {
// gather KPT patch values of this position, walking the reduction index incrementally
int kk = k0 + kc;
int kw = kk % args.KW;
int kh = (kk / args.KW) % args.KH;
int dx = kw*args.d0;
int dy = kh*args.d1;
// byte offset of the reduction index kk, relative to the patch origin
uint offk = (kk / (args.KW*args.KH))*nb12 + dy*nb11 + dx*nb10;
FOR_UNROLL (short i = 0; i < KPT; i++) {
const int ix = ix0 + dx;
const int iy = iy0 + dy;
half v = 0;
if (pok && kk < k_end && (uint) ix < (uint) args.IW && (uint) iy < (uint) args.IH) {
v = *(device const float *)(srcb + (off0 + offk));
}
sa[row*NK + kc + i] = v;
kk++;
dx += args.d0;
offk += args.d0*nb10;
if (++kw == args.KW) {
kw = 0;
dx = 0;
dy += args.d1;
offk += args.d1*nb11 - args.KW*args.d0*nb10;
if (++kh == args.KH) {
kh = 0;
dy = 0;
offk += nb12 - args.KH*args.d1*nb11;
}
}
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
// clamp the reduction extent so neither operand is read past the end of the split
const int kExt = min(NK, k_end - k0);
auto tAv = tensor(sa, dextents<int32_t, 2>(kExt, NRA), array<int, 2>({1, NK}));
auto tBv = tensor(ptrB + k0 + rb*strideB, dextents<int32_t, 2>(kExt, args.OC - rb), array<int, 2>({1, strideB}));
mm.run(tBv, tAv, cT);
threadgroup_barrier(mem_flags::mem_threadgroup);
}
// dst holds [OHW, OC] per image, with several splits it is the partial sum buffer, one contiguous output per split
const int tileM = min(NRA, OHW - ra);
const int tileN = min(NRB, args.OC - rb);
device float * dstTile = (device float *) (dst + (uint64_t) split*args.N*args.nb3 + img*args.nb3) + (uint64_t) rb*OHW + ra;
auto tD = tensor(dstTile, dextents<int32_t, 2>(tileM, tileN), array<int, 2>({1, OHW}));
cT.store(tD);
}
typedef decltype(kernel_conv_2d_mm_tensor<float>) kernel_conv_2d_mm_tensor_t;
template [[host_name("kernel_conv_2d_mm_tensor_f32_f32")]] kernel kernel_conv_2d_mm_tensor_t kernel_conv_2d_mm_tensor<float>;
template [[host_name("kernel_conv_2d_mm_tensor_f16_f32")]] kernel kernel_conv_2d_mm_tensor_t kernel_conv_2d_mm_tensor<half>;
#endif
kernel void kernel_conv_2d_mm_reduce(
constant ggml_metal_kargs_conv_2d_reduce & args,
device const float * tmp,
device float * dst,
uint tpig[[thread_position_in_grid]]) {
if ((int) tpig >= args.ne) {
return;
}
float sum = 0.0f;
for (int s = 0; s < args.nsplit; ++s) {
sum += tmp[(uint64_t) s*args.ne + tpig];
}
dst[tpig] = sum;
}
typedef decltype(kernel_conv_2d_mm<float, 64, 32>) kernel_conv_2d_mm_t;
template [[host_name("kernel_conv_2d_mm_f32_f32_64x32")]] kernel kernel_conv_2d_mm_t kernel_conv_2d_mm<float, 64, 32>;
+113 -6
View File
@@ -3307,6 +3307,85 @@ static int iq2_find_best_neighbour(const uint16_t * GGML_RESTRICT neighbours, co
return grid_index;
}
// same result as iq2_find_best_neighbour: score = s*(s*quad_coef - lin_coef) + const, and the coefs do not depend on the scale s, so cache them per pattern
// take the cached winner only if all other neighbours are worse by more than the float rounding bound of the exact search, else run the exact search
#define IQ2_NEIGHBOUR_CACHE_MAX 512
typedef struct {
int pattern; // pattern whose coefs are cached, -1 if none
int last_best; // last winner for this pattern
// sums for the rounding bound: |w|, |w|*|x|, |w|*x^2, and the same with 1 + |w|
double sum_w, sum_wx, sum_wxx, sum_w1, sum_w1x, sum_w1xx;
double lin_term[8][8], quad_term[8][8]; // 2*w*x*q and w*q*q per element, indexed by the grid level q
double lin_coef[IQ2_NEIGHBOUR_CACHE_MAX];
double quad_coef[IQ2_NEIGHBOUR_CACHE_MAX];
} iq2_neighbour_cache;
static void iq2_neighbour_cache_init(iq2_neighbour_cache * cache, const float * GGML_RESTRICT xval, const float * GGML_RESTRICT weight) {
cache->pattern = -1;
cache->sum_w = cache->sum_wx = cache->sum_wxx = cache->sum_w1 = cache->sum_w1x = cache->sum_w1xx = 0;
for (int i = 0; i < 8; ++i) {
const double w = (double)weight[i], x = (double)xval[i], abs_w = fabs(w), abs_x = fabs(x);
cache->sum_w += abs_w; cache->sum_wx += abs_w*abs_x; cache->sum_wxx += abs_w*abs_x*abs_x;
cache->sum_w1 += 1 + abs_w; cache->sum_w1x += (1 + abs_w)*abs_x; cache->sum_w1xx += (1 + abs_w)*abs_x*abs_x;
for (int q = 0; q < 8; ++q) {
cache->lin_term[i][q] = 2*w*x*q;
cache->quad_term[i][q] = w*q*q;
}
}
}
static int iq2_count_at_most(const double * GGML_RESTRICT values, int n, double limit) {
int count = 0;
for (int j = 0; j < n; ++j) count += values[j] <= limit;
return count;
}
static int iq2_find_best_neighbour_cached(iq2_neighbour_cache * cache, int pattern, const uint16_t * GGML_RESTRICT neighbours, const uint64_t * GGML_RESTRICT grid,
const float * GGML_RESTRICT xval, const float * GGML_RESTRICT weight, float scale, int8_t * GGML_RESTRICT L) {
const int num_neighbors = neighbours[0];
const double scale_d = (double)scale;
const double max_level = 7*scale_d;
// size of the numbers in the exact search, its rounding error is a multiple of 2^-24*size (+ size_subnormal for underflow)
const double size = max_level*max_level*cache->sum_w + 2*max_level*cache->sum_wx + cache->sum_wxx;
const double size_subnormal = (1 + max_level)*(1 + max_level)*cache->sum_w1 + 2*(1 + max_level)*cache->sum_w1x + cache->sum_w1xx;
// the cached score is (exact score - const)/s, so the 2^-18*size margin on the exact score is divided by s
const double tolerance = (0x1p-18*size + 0x1p-100*size_subnormal)/scale_d;
if (num_neighbors <= 0 || num_neighbors > IQ2_NEIGHBOUR_CACHE_MAX || !(scale_d > 0 && size < 0x1p100 && size_subnormal < 0x1p100 && tolerance > 0)) {
return iq2_find_best_neighbour(neighbours, grid, xval, weight, scale, L);
}
if (cache->pattern != pattern) {
for (int j = 0; j < num_neighbors; ++j) {
const int8_t * levels = (const int8_t *)(grid + neighbours[j+1]);
double lin = 0, quad = 0;
for (int i = 0; i < 8; ++i) {
lin += cache->lin_term[i][levels[i]];
quad += cache->quad_term[i][levels[i]];
}
cache->lin_coef[j] = lin; cache->quad_coef[j] = quad;
}
cache->pattern = pattern;
cache->last_best = 0;
}
double score[IQ2_NEIGHBOUR_CACHE_MAX];
for (int j = 0; j < num_neighbors; ++j) score[j] = scale_d*cache->quad_coef[j] - cache->lin_coef[j];
// the winner must be the only neighbour within tolerance, try the previous winner for this pattern first
int best = cache->last_best;
if (iq2_count_at_most(score, num_neighbors, score[best] + tolerance) != 1) {
for (int j = 0; j < num_neighbors; ++j) {
if (score[j] < score[best]) best = j;
}
if (iq2_count_at_most(score, num_neighbors, score[best] + tolerance) != 1) {
return iq2_find_best_neighbour(neighbours, grid, xval, weight, scale, L);
}
cache->last_best = best;
}
const int grid_index = neighbours[best+1];
const int8_t * levels = (const int8_t *)(grid + grid_index);
for (int i = 0; i < 8; ++i) L[i] = (levels[i] - 1)/2;
return grid_index;
}
static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML_RESTRICT vy, int64_t n, const float * GGML_RESTRICT quant_weights) {
const int gindex = iq2_data_index(GGML_TYPE_IQ2_XXS);
@@ -3335,6 +3414,8 @@ static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML
float waux[32];
uint8_t block_signs[4];
uint32_t q2[2*(QK_K/32)];
iq2_neighbour_cache neighbour_cache[4];
int8_t Laux_prev[32];
for (int ibl = 0; ibl < nbl; ++ibl) {
@@ -3390,6 +3471,7 @@ static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML
continue;
}
float best = 0;
for (int k = 0; k < 4; ++k) iq2_neighbour_cache_init(&neighbour_cache[k], xval + 8*k, waux + 8*k);
for (int is = -6; is <= 6; ++is) {
float id = (2*kMaxQ-1+is*0.1f)/eff_max;
float this_scale = 1/id;
@@ -3403,9 +3485,14 @@ static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML
int grid_index = kmap_q2xs[u];
if (grid_index < 0) {
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
}
}
// same candidate as for the previous scale: it cannot change the result
if (is > -6 && memcmp(Laux, Laux_prev, 32) == 0) {
continue;
}
memcpy(Laux_prev, Laux, 32);
float sumqx = 0, sumq2 = 0;
for (int i = 0; i < 32; ++i) {
float w = weight[i];
@@ -3430,7 +3517,7 @@ static void quantize_row_iq2_xxs_impl(const float * GGML_RESTRICT x, void * GGML
int grid_index = kmap_q2xs[u];
if (grid_index < 0) {
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
}
const int8_t * pg = (const int8_t *)(kgrid_q2xs + grid_index);
for (int i = 0; i < 8; ++i) L[8*k+i] = (pg[i] - 1)/2;
@@ -3515,6 +3602,9 @@ static void quantize_row_iq2_xs_impl(const float * GGML_RESTRICT x, void * GGML_
bool is_on_grid_aux[2];
uint8_t block_signs[2];
uint16_t q2[2*(QK_K/16)];
iq2_neighbour_cache neighbour_cache[2];
int8_t Laux_prev[16];
bool is_on_grid_prev[2];
for (int ibl = 0; ibl < nbl; ++ibl) {
@@ -3566,6 +3656,7 @@ static void quantize_row_iq2_xs_impl(const float * GGML_RESTRICT x, void * GGML_
float best = 0;
float scale = max/(2*kMaxQ-1);
is_on_grid[0] = is_on_grid[1] = true;
for (int k = 0; k < 2; ++k) iq2_neighbour_cache_init(&neighbour_cache[k], xval + 8*k, waux + 8*k);
for (int is = -9; is <= 9; ++is) {
float id = (2*kMaxQ-1+is*0.1f)/max;
float this_scale = 1/id;
@@ -3581,9 +3672,15 @@ static void quantize_row_iq2_xs_impl(const float * GGML_RESTRICT x, void * GGML_
if (grid_index < 0) {
is_on_grid_aux[k] = false;
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
}
}
// same candidate as for the previous scale: it cannot change the result
if (is > -9 && memcmp(Laux, Laux_prev, 16) == 0 && is_on_grid_aux[0] == is_on_grid_prev[0] && is_on_grid_aux[1] == is_on_grid_prev[1]) {
continue;
}
memcpy(Laux_prev, Laux, 16);
is_on_grid_prev[0] = is_on_grid_aux[0]; is_on_grid_prev[1] = is_on_grid_aux[1];
float sumqx = 0, sumq2 = 0;
for (int i = 0; i < 16; ++i) {
float w = weight[i];
@@ -3613,7 +3710,7 @@ static void quantize_row_iq2_xs_impl(const float * GGML_RESTRICT x, void * GGML_
int grid_index = kmap_q2xs[u];
if (grid_index < 0) {
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
}
}
float sumqx = 0, sumq2 = 0;
@@ -5155,6 +5252,9 @@ static void quantize_row_iq2_s_impl(const float * GGML_RESTRICT x, void * GGML_R
bool is_on_grid[2];
bool is_on_grid_aux[2];
uint8_t block_signs[2];
iq2_neighbour_cache neighbour_cache[2];
int8_t Laux_prev[16];
bool is_on_grid_prev[2];
for (int ibl = 0; ibl < nbl; ++ibl) {
@@ -5197,6 +5297,7 @@ static void quantize_row_iq2_s_impl(const float * GGML_RESTRICT x, void * GGML_R
float best = 0;
float scale = max/(2*kMaxQ-1);
is_on_grid[0] = is_on_grid[1] = true;
for (int k = 0; k < 2; ++k) iq2_neighbour_cache_init(&neighbour_cache[k], xval + 8*k, waux + 8*k);
for (int is = -9; is <= 9; ++is) {
float id = (2*kMaxQ-1+is*0.1f)/max;
float this_scale = 1/id;
@@ -5212,9 +5313,15 @@ static void quantize_row_iq2_s_impl(const float * GGML_RESTRICT x, void * GGML_R
if (grid_index < 0) {
is_on_grid_aux[k] = false;
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, this_scale, Laux + 8*k);
}
}
// same candidate as for the previous scale: it cannot change the result
if (is > -9 && memcmp(Laux, Laux_prev, 16) == 0 && is_on_grid_aux[0] == is_on_grid_prev[0] && is_on_grid_aux[1] == is_on_grid_prev[1]) {
continue;
}
memcpy(Laux_prev, Laux, 16);
is_on_grid_prev[0] = is_on_grid_aux[0]; is_on_grid_prev[1] = is_on_grid_aux[1];
float sumqx = 0, sumq2 = 0;
for (int i = 0; i < 16; ++i) {
float w = weight[i];
@@ -5244,7 +5351,7 @@ static void quantize_row_iq2_s_impl(const float * GGML_RESTRICT x, void * GGML_R
int grid_index = kmap_q2xs[u];
if (grid_index < 0) {
const uint16_t * neighbours = kneighbors_q2xs - kmap_q2xs[u] - 1;
grid_index = iq2_find_best_neighbour(neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
grid_index = iq2_find_best_neighbour_cached(&neighbour_cache[k], u, neighbours, kgrid_q2xs, xval + 8*k, waux + 8*k, scale, L + 8*k);
}
}
float sumqx = 0, sumq2 = 0;
+1 -1
View File
@@ -101,7 +101,7 @@ void deferred_memset(void * dst, uint32_t val, size_t size, std::vector<vk_stagi
// matmul
vk_pipeline ggml_vk_get_to_fp16(ggml_backend_vk_context * ctx, ggml_type type);
void ggml_vk_matmul(ggml_backend_vk_context * ctx, vk_context& subctx, vk_pipeline& pipeline, vk_subbuffer&& a, vk_subbuffer&& b, vk_subbuffer&& d, vk_subbuffer&& split_k_buffer, uint32_t m, uint32_t n, uint32_t k, uint32_t stride_a, uint32_t stride_b, uint32_t stride_d, uint32_t batch_stride_a, uint32_t batch_stride_b, uint32_t batch_stride_d, uint32_t split_k, uint32_t batch, uint32_t ne02, uint32_t ne12, uint32_t broadcast2, uint32_t broadcast3, uint32_t padded_n, uint64_t b_col_bytes, uint64_t d_col_bytes);
void ggml_vk_matmul(ggml_backend_vk_context * ctx, vk_context& subctx, vk_pipeline& pipeline, vk_subbuffer&& a, vk_subbuffer&& b, vk_subbuffer&& d, vk_subbuffer&& split_k_buffer, uint32_t m, uint32_t n, uint32_t k, uint32_t stride_a, uint32_t stride_b, uint32_t stride_d, uint32_t batch_stride_a, uint32_t batch_stride_b, uint32_t batch_stride_d, uint32_t split_k, uint32_t stream_k_groups, uint32_t batch, uint32_t ne02, uint32_t ne12, uint32_t broadcast2, uint32_t broadcast3, uint32_t padded_n, uint64_t b_col_bytes, uint64_t d_col_bytes);
bool ggml_vk_dim01_contiguous(const ggml_tensor * tensor);
vk_pipeline ggml_vk_get_cpy_pipeline(ggml_backend_vk_context * ctx, const ggml_tensor * src, const ggml_tensor * dst, ggml_type to);
vk_pipeline ggml_vk_get_quantize_pipeline(ggml_backend_vk_context * ctx, ggml_type type);
+5 -3
View File
@@ -225,7 +225,7 @@ void ggml_vk_test_matmul(ggml_backend_vk_context * ctx, size_t m, size_t n, size
ctx, subctx, p, ggml_vk_subbuffer(ctx, d_X), ggml_vk_subbuffer(ctx, d_Y), ggml_vk_subbuffer(ctx, d_D), ggml_vk_subbuffer(ctx, ctx->prealloc_split_k),
m, n, k,
k, k, m, k*m, k*n, m*n,
split_k, batch, batch, batch, 1, 1, n,
split_k, 0, batch, batch, batch, 1, 1, n,
k * sizeof(Y_TYPE), m * sizeof(float)
);
}
@@ -693,7 +693,7 @@ void ggml_vk_test_dequant_matmul(ggml_backend_vk_context * ctx, size_t m, size_t
ctx, subctx, p, { qx_buf, 0, qx_sz }, { qy_buf, 0, qy_sz }, { d_buf, 0, d_sz }, { ctx->prealloc_split_k, 0, ctx->prealloc_size_split_k },
m, n, k,
k, k, m, k*m, k*n, m*n,
split_k, batch, batch, batch, 1, 1, n,
split_k, 0, batch, batch, batch, 1, 1, n,
(uint64_t)(k / ggml_blck_size(GGML_TYPE_Q8_1)) * ggml_type_size(GGML_TYPE_Q8_1), m * sizeof(float)
);
}
@@ -703,7 +703,7 @@ void ggml_vk_test_dequant_matmul(ggml_backend_vk_context * ctx, size_t m, size_t
ctx, subctx, p, { qx_buf, 0, qx_sz }, { y_buf, 0, y_sz }, { d_buf, 0, d_sz }, { ctx->prealloc_split_k, 0, ctx->prealloc_size_split_k },
m, n, k,
k, k, m, k*m, k*n, m*n,
split_k, batch, batch, batch, 1, 1, n,
split_k, 0, batch, batch, batch, 1, 1, n,
k * sizeof(float), m * sizeof(float)
);
}
@@ -1043,6 +1043,8 @@ void ggml_vk_check_results_0(ggml_backend_vk_context * ctx, ggml_cgraph * cgraph
} else if (tensor->op == GGML_OP_FILL) {
const float value = ggml_get_op_params_f32(tensor, 0);
tensor_clone = ggml_fill(ggml_ctx, src_clone[0], value);
} else if (tensor->op == GGML_OP_SLEEP) {
tensor_clone = ggml_sleep(ggml_ctx, src_clone[0], ggml_get_op_params_i32(tensor, 0));
} else if (tensor->op == GGML_OP_SQR) {
tensor_clone = ggml_sqr(ggml_ctx, src_clone[0]);
} else if (tensor->op == GGML_OP_SQRT) {
@@ -13,8 +13,38 @@ struct vk_mat_mat_push_constants {
uint32_t k_split;
uint32_t ne02; uint32_t ne12; uint32_t broadcast2; uint32_t broadcast3;
uint32_t padded_N;
uint32_t k_units_per_tile;
uint32_t work_units_per_group;
uint32_t groups_with_extra_unit;
uint32_t k_units_mp;
uint32_t blocks_m_mp;
uint32_t blocks_n_mp;
uint32_t ne12_mp;
uint32_t broadcast2_mp;
uint32_t broadcast3_mp;
uint32_t div_Ls0;
uint32_t div_Ls1;
};
struct vk_mat_mat_stream_k_fixup_push_constants {
uint32_t M; uint32_t N;
uint32_t BM; uint32_t BN;
uint32_t k_units_per_tile;
uint32_t blocks_m;
uint32_t blocks_n;
uint32_t work_units_per_group;
uint32_t groups_with_extra_unit;
uint32_t k_units_mp;
uint32_t blocks_m_mp;
uint32_t blocks_n_mp;
uint32_t work_units_mp;
uint32_t large_group_work_mp;
uint32_t div_Ls0;
uint32_t div_Ls1;
};
static_assert(sizeof(vk_mat_mat_push_constants) <= 128);
struct vk_mat_vec_push_constants {
uint32_t ncols;
uint32_t stride_a;
@@ -159,6 +189,10 @@ struct vk_op_push_constants {
float param4;
};
struct vk_op_sleep_push_constants {
uint32_t ticks;
};
struct vk_op_fwht_push_constants {
uint32_t n_rows;
uint32_t src_offset;
@@ -403,6 +437,34 @@ template <typename T> void init_pushconst_fastdiv(T &p) {
static_assert(!std::is_const<T>::value, "unexpected type");
}
inline void init_pushconst_fastdiv(vk_mat_mat_push_constants &p) {
uint32_t ne12_L, broadcast2_L, broadcast3_L;
init_fastdiv_values(p.ne12, p.ne12_mp, ne12_L);
init_fastdiv_values(p.broadcast2, p.broadcast2_mp, broadcast2_L);
init_fastdiv_values(p.broadcast3, p.broadcast3_mp, broadcast3_L);
p.div_Ls1 = pack_fastdiv_L(ne12_L, broadcast2_L, broadcast3_L);
}
inline void init_pushconst_fastdiv(vk_mat_mat_push_constants &p, uint32_t blocks_m, uint32_t blocks_n) {
uint32_t k_units_L, blocks_m_L, blocks_n_L;
init_fastdiv_values(p.k_units_per_tile, p.k_units_mp, k_units_L);
init_fastdiv_values(blocks_m, p.blocks_m_mp, blocks_m_L);
init_fastdiv_values(blocks_n, p.blocks_n_mp, blocks_n_L);
p.div_Ls0 = pack_fastdiv_L(k_units_L, blocks_m_L, blocks_n_L);
init_pushconst_fastdiv(p);
}
template <> inline void init_pushconst_fastdiv(vk_mat_mat_stream_k_fixup_push_constants &p) {
uint32_t k_units_L, blocks_m_L, blocks_n_L, work_units_L, large_group_work_L;
init_fastdiv_values(p.k_units_per_tile, p.k_units_mp, k_units_L);
init_fastdiv_values(p.blocks_m, p.blocks_m_mp, blocks_m_L);
init_fastdiv_values(p.blocks_n, p.blocks_n_mp, blocks_n_L);
init_fastdiv_values(p.work_units_per_group, p.work_units_mp, work_units_L);
init_fastdiv_values(p.work_units_per_group + 1, p.large_group_work_mp, large_group_work_L);
p.div_Ls0 = pack_fastdiv_L(k_units_L, blocks_m_L, blocks_n_L);
p.div_Ls1 = pack_fastdiv_L(work_units_L, large_group_work_L, 0);
}
template <> inline void init_pushconst_fastdiv(vk_op_unary_push_constants &p) {
// Compute magic values to divide by these six numbers.
uint32_t ne0_012L;
+13 -2
View File
@@ -259,10 +259,11 @@ struct vk_matmul_pipeline_key {
ggml_type type_b;
bool mul_mat_id;
bool f16acc;
bool stream_k = false;
bool operator<(const vk_matmul_pipeline_key & o) const {
return std::tie(type_a, type_b, mul_mat_id, f16acc)
< std::tie(o.type_a, o.type_b, o.mul_mat_id, o.f16acc);
return std::tie(type_a, type_b, mul_mat_id, f16acc, stream_k)
< std::tie(o.type_a, o.type_b, o.mul_mat_id, o.f16acc, o.stream_k);
}
};
@@ -738,6 +739,7 @@ struct vk_device_struct {
bool shader_int64;
bool buffer_device_address;
bool vulkan_memory_model;
bool shader_clock;
bool add_rms_fusion;
uint32_t partials_binding_alignment;
@@ -746,6 +748,10 @@ struct vk_device_struct {
bool shader_64b_indexing;
bool integer_dot_product;
// Which 4x8 bit packed integer dot products are accelerated
bool integer_dot_signed {};
bool integer_dot_mixed {};
bool integer_dot_unsigned {};
// 0: default, 1: force mmvq, -1: disable mmvq
int32_t mmvq_mode;
@@ -819,6 +825,7 @@ struct vk_device_struct {
matmul_tile_selector_t matmul_id_tile_selector;
vk_pipeline pipeline_matmul_split_k_reduce;
vk_pipeline pipeline_matmul_stream_k_fixup;
vk_pipeline pipeline_quantize_q8_1_x4;
vk_pipeline pipeline_dequant[GGML_TYPE_COUNT];
@@ -917,6 +924,8 @@ struct vk_device_struct {
vk_pipeline pipeline_fill_f32;
vk_pipeline pipeline_fill_f16;
vk_pipeline pipeline_sleep;
// [src/dst 0=fp32,1=fp16,2=bf16]
vk_pipeline pipeline_glu[GGML_GLU_OP_COUNT][3];
@@ -1023,6 +1032,8 @@ struct vk_device_struct {
std::map<std::string, uint64_t> fusion_counts;
bool disable_descriptor_reuse;
std::atomic<uint64_t> buffer_destroy_count {};
bool disable_stream_k;
bool force_stream_k;
bool disable_host_visible_vidmem;
bool allow_sysmem_fallback;
bool disable_graph_optimize;
+281 -51
View File
@@ -1726,6 +1726,11 @@ static bool ggml_vk_matmul_shmem_support(const vk_device& device, const std::vec
return supported;
}
// MMQ uses the mixed signedness dot product (MMQ_SUDOT) when the signed one is not accelerated
static bool ggml_vk_mmq_use_sudot(const vk_device& device) {
return device->integer_dot_mixed && !device->integer_dot_signed;
}
static bool ggml_vk_matmul_int_shmem_support(const vk_device& device, const std::vector<uint32_t>& warptile, bool mul_mat_id, ggml_type src0_type) {
// FLOAT_TYPE in the shader is float16_t with fp16 support, otherwise float.
@@ -1782,7 +1787,12 @@ static bool ggml_vk_matmul_int_shmem_support(const vk_device& device, const std:
const uint32_t buf_b_size = BN * BK_STEP * block_b_size;
const uint32_t mmid_row_ids = mul_mat_id ? (BN * 2u * (uint32_t)sizeof(uint16_t)) : 0u;
const uint32_t total_size = buf_a_size + buf_b_size + mmid_row_ids + lut_size;
// Signed quants declare buf_b_qsum: 2 int32_t per B block with MMQ_SUDOT, a single int32_t without
const bool b_sums = src0_type == GGML_TYPE_Q8_0 || src0_type == GGML_TYPE_MXFP4 || src0_type == GGML_TYPE_IQ4_XS ||
src0_type == GGML_TYPE_IQ3_S || src0_type == GGML_TYPE_Q3_K || src0_type == GGML_TYPE_Q6_K;
const uint32_t qsum_size = !b_sums ? 0u : ggml_vk_mmq_use_sudot(device) ? BN * BK_STEP * 2u * 4u : 4u;
const uint32_t total_size = buf_a_size + buf_b_size + mmid_row_ids + lut_size + qsum_size;
const bool supported = total_size <= device->properties.limits.maxComputeSharedMemorySize;
VK_LOG_DEBUG("ggml_vk_matmul_int_shmem_support(warptile=(" << warptile[0] << "," << warptile[1] << "," << warptile[2] << "), "
@@ -1906,7 +1916,7 @@ static bool ggml_vk_fa_type_needs_shmem(ggml_type type) {
static bool ggml_vk_fa_scalar_uses_mmq(const vk_device& device, ggml_type k_type, ggml_type v_type) {
#if defined(GGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT)
return device->integer_dot_product && device->subgroup_clustered &&
return device->integer_dot_product && device->integer_dot_signed && device->subgroup_clustered &&
!ggml_vk_fa_type_needs_shmem(v_type) &&
(k_type == GGML_TYPE_Q4_0 || k_type == GGML_TYPE_Q4_1 ||
k_type == GGML_TYPE_Q5_0 || k_type == GGML_TYPE_Q5_1 ||
@@ -2467,6 +2477,7 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
FOR_EACH_LUT_FP4_TYPE(X)
const int mul_mat_id_param_count = 5;
constexpr uint32_t stream_k_spec_id = 15;
using spec_fn_t = std::function<std::vector<uint32_t>(const std::vector<uint32_t>&, bool)>;
auto const &create_mm_pipelines = [&](
@@ -2478,6 +2489,11 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
bool disable_robustness = false, bool require_full_subgroups = false, uint32_t required_subgroup_size = 0,
bool create_aligned = true, bool pin_subgroup_to_warp = false
) {
GGML_ASSERT(!key.stream_k || (!key.mul_mat_id && param_count == 3 && push_constant_size == sizeof(vk_mat_mat_push_constants) && create_aligned));
// Non-ID matmul shaders declare the Stream-K scratch binding in all specializations.
if (!key.mul_mat_id) {
++param_count;
}
auto& vec = device->pipeline_matmul[key];
const bool first_call = vec.empty();
for (size_t i = 0; i < tile_configs.size(); i++) {
@@ -2495,25 +2511,40 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
vk_matmul_pipeline_pair pair{};
pair.align = tc.align;
pair.narrow_n_max = tc.narrow_n_max;
std::string suffix = "_" + std::to_string(i);
pair.unaligned = std::make_shared<vk_pipeline_struct>();
if (!key.stream_k) {
pair.unaligned = std::make_shared<vk_pipeline_struct>();
}
if (create_aligned) {
pair.aligned = std::make_shared<vk_pipeline_struct>();
}
if (key.stream_k) {
// Tile selection reads unaligned even when selecting an aligned pipeline.
pair.unaligned = pair.aligned;
}
vec.push_back(pair);
}
ggml_vk_create_pipeline(device, vec[i].unaligned,
vec[i].unaligned->name.empty() ? (shader_name + "_" + std::to_string(i)).c_str() : vec[i].unaligned->name.c_str(),
spv_len, spv_data, "main", param_count, push_constant_size,
tc.wg_denoms, spec_fn(tc.warptile, false), 1,
disable_robustness, rfs, rsgs);
if (!key.stream_k) {
ggml_vk_create_pipeline(device, vec[i].unaligned,
vec[i].unaligned->name.empty() ? (shader_name + "_" + std::to_string(i)).c_str() : vec[i].unaligned->name.c_str(),
spv_len, spv_data, "main", param_count, push_constant_size,
tc.wg_denoms, spec_fn(tc.warptile, false), 1,
disable_robustness, rfs, rsgs);
}
if (vec[i].aligned) {
auto spec = spec_fn(tc.warptile, true);
if (key.stream_k) {
if (spec.size() <= stream_k_spec_id) {
spec.resize(stream_k_spec_id + 1);
}
spec[stream_k_spec_id] = 1;
}
const std::string name = shader_name + (key.stream_k ? "_stream_k_" : "_aligned_") + std::to_string(i);
ggml_vk_create_pipeline(device, vec[i].aligned,
vec[i].aligned->name.empty() ? (shader_name + "_aligned_" + std::to_string(i)).c_str() : vec[i].aligned->name.c_str(),
vec[i].aligned->name.empty() ? name.c_str() : vec[i].aligned->name.c_str(),
spv_len, spv_data, "main", param_count, push_constant_size,
tc.wg_denoms, spec_fn(tc.warptile, true), tc.align,
tc.wg_denoms, spec, tc.align,
disable_robustness, rfs, rsgs);
}
}
@@ -2611,13 +2642,19 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
}
spec_fn_t cm2_spec = [&](const std::vector<uint32_t>& wt, bool a) { return ggml_vk_mul_mm_cm2_spec(wt, a); };
auto create_cm2_pipelines = [&](vk_matmul_pipeline_key key, const std::vector<vk_tile_config>& tc,
const std::string& name, size_t len, const void* data, uint32_t pc_size, uint32_t pc, const spec_fn_t& spec) {
create_mm_pipelines(key, tc, name, len, data, pc_size, pc, spec, true);
key.stream_k = true;
create_mm_pipelines(key, tc, name, len, data, pc_size, pc, spec, true);
};
// F16 x F16
create_mm_pipelines({GGML_TYPE_F16, GGML_TYPE_F16, false, true}, tc_mm, "matmul_f16_f16acc", matmul_f16_f16acc_cm2_len, matmul_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_mm_pipelines({GGML_TYPE_F16, GGML_TYPE_F16, false, false}, tc_mm, "matmul_f16", matmul_f16_cm2_len, matmul_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_cm2_pipelines({GGML_TYPE_F16, GGML_TYPE_F16, false, true}, tc_mm, "matmul_f16_f16acc", matmul_f16_f16acc_cm2_len, matmul_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
create_cm2_pipelines({GGML_TYPE_F16, GGML_TYPE_F16, false, false}, tc_mm, "matmul_f16", matmul_f16_cm2_len, matmul_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
#if defined(GGML_VULKAN_BFLOAT16_GLSLC_SUPPORT)
if (device->coopmat_bf16_support) {
create_mm_pipelines({GGML_TYPE_BF16, GGML_TYPE_BF16, false, false}, tc_mm, "matmul_bf16", matmul_bf16_cm2_len, matmul_bf16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_cm2_pipelines({GGML_TYPE_BF16, GGML_TYPE_BF16, false, false}, tc_mm, "matmul_bf16", matmul_bf16_cm2_len, matmul_bf16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
}
#endif
for (const auto type : non_lut_quant_types) {
@@ -2627,18 +2664,18 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
}
auto& tc = ((type >= GGML_TYPE_Q2_K && type <= GGML_TYPE_Q6_K) || type == GGML_TYPE_TQ1_0 || type == GGML_TYPE_TQ2_0) ? tc_mmq_k : tc_mmq;
spec_fn_t qs = [&, type](const std::vector<uint32_t>& wt, bool a) { return ggml_vk_mul_mm_cm2_spec(wt, a, (uint32_t)type); };
create_mm_pipelines({type, GGML_TYPE_F16, false, true}, tc, "matmul_quant_f16_f16acc", matmul_quant_f16_f16acc_cm2_len, matmul_quant_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, qs, true);
create_mm_pipelines({type, GGML_TYPE_F16, false, false}, tc, "matmul_quant_f16", matmul_quant_f16_cm2_len, matmul_quant_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, qs, true);
create_cm2_pipelines({type, GGML_TYPE_F16, false, true}, tc, "matmul_quant_f16_f16acc", matmul_quant_f16_f16acc_cm2_len, matmul_quant_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, qs);
create_cm2_pipelines({type, GGML_TYPE_F16, false, false}, tc, "matmul_quant_f16", matmul_quant_f16_cm2_len, matmul_quant_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, qs);
}
create_mm_pipelines({GGML_TYPE_Q4_K, GGML_TYPE_F16, false, true}, tc_mmq_k, "matmul_q4_k_f16_f16acc", matmul_q4_k_f16_f16acc_cm2_len, matmul_q4_k_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_mm_pipelines({GGML_TYPE_Q4_K, GGML_TYPE_F16, false, false}, tc_mmq_k, "matmul_q4_k_f16", matmul_q4_k_f16_cm2_len, matmul_q4_k_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_mm_pipelines({GGML_TYPE_Q5_K, GGML_TYPE_F16, false, true}, tc_mmq_k, "matmul_q5_k_f16_f16acc", matmul_q5_k_f16_f16acc_cm2_len, matmul_q5_k_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_mm_pipelines({GGML_TYPE_Q5_K, GGML_TYPE_F16, false, false}, tc_mmq_k, "matmul_q5_k_f16", matmul_q5_k_f16_cm2_len, matmul_q5_k_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true);
create_cm2_pipelines({GGML_TYPE_Q4_K, GGML_TYPE_F16, false, true}, tc_mmq_k, "matmul_q4_k_f16_f16acc", matmul_q4_k_f16_f16acc_cm2_len, matmul_q4_k_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
create_cm2_pipelines({GGML_TYPE_Q4_K, GGML_TYPE_F16, false, false}, tc_mmq_k, "matmul_q4_k_f16", matmul_q4_k_f16_cm2_len, matmul_q4_k_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
create_cm2_pipelines({GGML_TYPE_Q5_K, GGML_TYPE_F16, false, true}, tc_mmq_k, "matmul_q5_k_f16_f16acc", matmul_q5_k_f16_f16acc_cm2_len, matmul_q5_k_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
create_cm2_pipelines({GGML_TYPE_Q5_K, GGML_TYPE_F16, false, false}, tc_mmq_k, "matmul_q5_k_f16", matmul_q5_k_f16_cm2_len, matmul_q5_k_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec);
#define X_CM2(TYPE, tstr) \
{ auto tc = filter_tc(tc_mmq, TYPE, false); \
if (!tc.empty()) { \
create_mm_pipelines({TYPE, GGML_TYPE_F16, false, true}, tc, "matmul_" #tstr "_f16_f16acc", matmul_##tstr##_f16_f16acc_cm2_len, matmul_##tstr##_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
create_mm_pipelines({TYPE, GGML_TYPE_F16, false, false}, tc, "matmul_" #tstr "_f16", matmul_##tstr##_f16_cm2_len, matmul_##tstr##_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
create_cm2_pipelines({TYPE, GGML_TYPE_F16, false, true}, tc, "matmul_" #tstr "_f16_f16acc", matmul_##tstr##_f16_f16acc_cm2_len, matmul_##tstr##_f16_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec); \
create_cm2_pipelines({TYPE, GGML_TYPE_F16, false, false}, tc, "matmul_" #tstr "_f16", matmul_##tstr##_f16_cm2_len, matmul_##tstr##_f16_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec); \
} }
FOR_EACH_LUT_TYPE_NONFP4(X_CM2)
#if defined(GGML_VULKAN_FLOAT_E2M1_GLSLC_SUPPORT) && defined(GGML_VULKAN_FLOAT_E4M3_GLSLC_SUPPORT)
@@ -2646,8 +2683,8 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
#define X_CM2_OCP(TYPE, tstr) \
{ auto tc = filter_tc(tc_mmq, TYPE, false); \
if (!tc.empty()) { \
create_mm_pipelines({TYPE, GGML_TYPE_F16, false, true}, tc, "matmul_" #tstr "_f16_ocp_f16acc", matmul_##tstr##_f16_ocp_f16acc_cm2_len, matmul_##tstr##_f16_ocp_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
create_mm_pipelines({TYPE, GGML_TYPE_F16, false, false}, tc, "matmul_" #tstr "_f16_ocp", matmul_##tstr##_f16_ocp_cm2_len, matmul_##tstr##_f16_ocp_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec, true); \
create_cm2_pipelines({TYPE, GGML_TYPE_F16, false, true}, tc, "matmul_" #tstr "_f16_ocp_f16acc", matmul_##tstr##_f16_ocp_f16acc_cm2_len, matmul_##tstr##_f16_ocp_f16acc_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec); \
create_cm2_pipelines({TYPE, GGML_TYPE_F16, false, false}, tc, "matmul_" #tstr "_f16_ocp", matmul_##tstr##_f16_ocp_cm2_len, matmul_##tstr##_f16_ocp_cm2_data, sizeof(vk_mat_mat_push_constants), 3, cm2_spec); \
} }
FOR_EACH_LUT_FP4_TYPE(X_CM2_OCP)
#undef X_CM2_OCP
@@ -2925,8 +2962,13 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
uint32_t rsgs = 0) {
auto tc = filter_tc(tc_base, key.type_a, key.mul_mat_id, true);
if (!tc.empty()) {
spec_fn_t identity = [](const std::vector<uint32_t>& wt, bool) { return wt; };
create_mm_pipelines(key, tc, name, len, data, pc_size, pc, identity, false, rsgs > 0, rsgs, false);
// constant_id 11 is MMQ_SUDOT
spec_fn_t mmq_spec = [&](const std::vector<uint32_t>& wt, bool) {
std::vector<uint32_t> spec = wt;
spec.push_back(ggml_vk_mmq_use_sudot(device) ? 1u : 0u);
return spec;
};
create_mm_pipelines(key, tc, name, len, data, pc_size, pc, mmq_spec, false, rsgs > 0, rsgs, false);
}
};
#endif
@@ -3515,6 +3557,7 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
ggml_vk_create_pipeline(device, device->pipeline_get_rows_back_f32, "get_rows_back_f32", get_rows_back_f32_len, get_rows_back_f32_data, "main", 3, sizeof(vk_op_binary_push_constants), {256, 1, 1}, {}, 1, true);
ggml_vk_create_pipeline(device, device->pipeline_matmul_split_k_reduce, "split_k_reduce", split_k_reduce_len, split_k_reduce_data, "main", 2, 2 * sizeof(uint32_t), {256 * 4, 1, 1}, {}, 1);
ggml_vk_create_pipeline(device, device->pipeline_matmul_stream_k_fixup, "stream_k_fixup", stream_k_fixup_len, stream_k_fixup_data, "main", 2, sizeof(vk_mat_mat_stream_k_fixup_push_constants), {1, 32, 1}, {}, 1);
ggml_vk_create_pipeline(device, device->pipeline_flash_attn_split_k_reduce, "fa_split_k_reduce", fa_split_k_reduce_len, fa_split_k_reduce_data, "main", 3, sizeof(vk_op_flash_attn_split_k_reduce_push_constants), {1, device->subgroup_size, 1}, {device->subgroup_size}, 1, true);
#if defined(VK_KHR_cooperative_matrix) && defined(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
@@ -3849,6 +3892,10 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
ggml_vk_create_pipeline(device, device->pipeline_fill_f32, "fill_f32", fill_f32_len, fill_f32_data, "main", 1, sizeof(vk_op_push_constants), {512, 1, 1}, {}, 1);
ggml_vk_create_pipeline(device, device->pipeline_fill_f16, "fill_f16", fill_f16_len, fill_f16_data, "main", 1, sizeof(vk_op_push_constants), {512, 1, 1}, {}, 1);
if (device->shader_clock) {
ggml_vk_create_pipeline(device, device->pipeline_sleep, "sleep", sleep_len, sleep_data, "main", 1, sizeof(vk_op_sleep_push_constants), {1, 1, 1}, {}, 1);
}
#define CREATE_GLU(name, op) \
ggml_vk_create_pipeline(device, device->pipeline_glu[GGML_GLU_OP_ ## op][0], #name "_f32", name ## _f32_len, name ## _f32_data, "main", 3, sizeof(vk_op_glu_push_constants), {512, 1, 1}, {}, 1, true); \
ggml_vk_create_pipeline(device, device->pipeline_glu[GGML_GLU_OP_ ## op][1], #name "_f16", name ## _f16_len, name ## _f16_data, "main", 3, sizeof(vk_op_glu_push_constants), {512, 1, 1}, {}, 1, true); \
@@ -4478,6 +4525,7 @@ vk_device ggml_vk_get_device(size_t idx) {
bool ocp_microscaling_extension = false;
bool shader_float8_extension = false;
bool workgroup_memory_explicit_layout_support = false;
bool shader_clock_support = false;
for (const auto& properties : ext_props) {
if (strcmp("VK_KHR_maintenance4", properties.extensionName) == 0) {
@@ -4528,6 +4576,9 @@ vk_device ggml_vk_get_device(size_t idx) {
} else if (strcmp(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME, properties.extensionName) == 0) {
shader_float8_extension = true;
#endif
} else if (strcmp("VK_KHR_shader_clock", properties.extensionName) == 0 &&
!getenv("GGML_VK_DISABLE_SHADER_CLOCK")) {
shader_clock_support = true;
} else if (strcmp("VK_VALVE_shader_mixed_float_dot_product", properties.extensionName) == 0 &&
!getenv("GGML_VK_DISABLE_DOT2")) {
dot2_f16_support = true;
@@ -4716,7 +4767,10 @@ vk_device ggml_vk_get_device(size_t idx) {
device->coopmat_support = false;
}
device->integer_dot_product = device->integer_dot_product && shader_integer_dot_product_props.integerDotProduct4x8BitPackedSignedAccelerated;
device->integer_dot_signed = shader_integer_dot_product_props.integerDotProduct4x8BitPackedSignedAccelerated;
device->integer_dot_mixed = shader_integer_dot_product_props.integerDotProduct4x8BitPackedMixedSignednessAccelerated;
device->integer_dot_unsigned = shader_integer_dot_product_props.integerDotProduct4x8BitPackedUnsignedAccelerated;
device->integer_dot_product = device->integer_dot_product && (device->integer_dot_signed || device->integer_dot_mixed);
device->min_imported_host_pointer_alignment = external_memory_host_props.minImportedHostPointerAlignment;
@@ -4873,6 +4927,14 @@ vk_device ggml_vk_get_device(size_t idx) {
device_extensions.push_back("VK_KHR_shader_integer_dot_product");
}
VkPhysicalDeviceShaderClockFeaturesKHR shader_clock_features {};
shader_clock_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_CLOCK_FEATURES_KHR;
if (shader_clock_support) {
last_struct->pNext = (VkBaseOutStructure *)&shader_clock_features;
last_struct = (VkBaseOutStructure *)&shader_clock_features;
device_extensions.push_back("VK_KHR_shader_clock");
}
VkPhysicalDeviceShaderMixedFloatDotProductFeaturesVALVE dot2_features {};
dot2_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_MIXED_FLOAT_DOT_PRODUCT_FEATURES_VALVE;
if (dot2_f16_support) {
@@ -4979,6 +5041,7 @@ vk_device ggml_vk_get_device(size_t idx) {
device->shader_int64 = device_features2.features.shaderInt64;
device->buffer_device_address = vk12_features.bufferDeviceAddress;
device->vulkan_memory_model = vk12_features.vulkanMemoryModel;
device->shader_clock = shader_clock_support && shader_clock_features.shaderDeviceClock;
if (device->subgroup_size_control) {
device->subgroup_min_size = subgroup_size_control_props.minSubgroupSize;
@@ -5375,6 +5438,8 @@ vk_device ggml_vk_get_device(size_t idx) {
device->serialize_submissions = getenv("GGML_VK_SERIALIZE_SUBMISSIONS") != nullptr;
device->disable_fusion = getenv("GGML_VK_DISABLE_FUSION") != nullptr;
device->disable_stream_k = getenv("GGML_VK_DISABLE_STREAM_K") != nullptr;
device->force_stream_k = getenv("GGML_VK_FORCE_STREAM_K") != nullptr;
device->disable_descriptor_reuse = getenv("GGML_VK_DISABLE_DESCRIPTOR_REUSE") != nullptr;
@@ -5593,7 +5658,8 @@ static void ggml_vk_print_gpu_info(size_t idx) {
const bool rebar = ggml_vk_has_rebar(physical_device);
integer_dot_product = integer_dot_product
&& shader_integer_dot_product_props.integerDotProduct4x8BitPackedSignedAccelerated
&& (shader_integer_dot_product_props.integerDotProduct4x8BitPackedSignedAccelerated ||
shader_integer_dot_product_props.integerDotProduct4x8BitPackedMixedSignednessAccelerated)
&& shader_integer_dot_product_features.shaderIntegerDotProduct;
coopmat_support = coopmat_support
@@ -6408,14 +6474,113 @@ static uint32_t ggml_vk_guess_split_k(ggml_backend_vk_context * ctx, uint32_t m,
return split_k;
}
static uint32_t ggml_vk_guess_stream_k_groups(
ggml_backend_vk_context * ctx, const vk_pipeline & pipeline,
uint32_t m, uint32_t n, uint32_t k, uint32_t stride_d, uint32_t batch, uint32_t split_k, bool disable_split_k) {
if (disable_split_k) {
return 0;
}
if (ctx->device->disable_stream_k) {
return 0;
}
// TODO: Tune the CM1 and scalar workgroup counts before enabling Stream-K.
if (!ctx->device->coopmat2) {
return 0;
}
if (ctx->device->shader_core_count == 0) {
return 0;
}
const uint32_t bm = pipeline->wg_denoms[0];
const uint32_t bn = pipeline->wg_denoms[1];
if (bm % 32 != 0 || m % bm != 0 || n % bn != 0 || k % 256 != 0 || stride_d != m) {
return 0;
}
const uint32_t k_units = k / 256;
if (k_units == 1) {
return 0;
}
const uint64_t tiles = uint64_t(m / bm) * (n / bn) * batch;
const uint64_t total_work_units = tiles * k_units;
if (total_work_units > UINT32_MAX) {
return 0;
}
// Coopmat2 tiles are tuned for one resident workgroup per SM.
const uint32_t target_groups = ctx->device->shader_core_count;
const uint64_t regular_groups = tiles * split_k;
const uint64_t regular_waves = CEIL_DIV(regular_groups, target_groups);
const uint64_t regular_capacity = regular_waves * target_groups;
const uint32_t stream_k_groups = std::min<uint64_t>(target_groups, total_work_units);
if (stream_k_groups > ctx->device->properties.limits.maxComputeWorkGroupCount[0] ||
bm / 32 > ctx->device->properties.limits.maxComputeWorkGroupCount[1]) {
return 0;
}
if (!ctx->device->force_stream_k) {
if (regular_waves == 1 && stream_k_groups == regular_groups) {
return 0;
}
// Stream-K does not offset its overhead when the regular grid fills most of the last wave.
// Use a lower cutoff for short K with no K split and narrow N tiles.
if ((split_k == 1 && k <= 4096 && bn <= 128 && regular_groups * 4 >= regular_capacity * 3) ||
regular_groups * 10 >= regular_capacity * 9) {
return 0;
}
}
return stream_k_groups;
}
void ggml_vk_matmul(
ggml_backend_vk_context * ctx, vk_context& subctx, vk_pipeline& pipeline,
vk_subbuffer&& a, vk_subbuffer&& b, vk_subbuffer&& d, vk_subbuffer&& split_k_buffer,
uint32_t m, uint32_t n, uint32_t k, uint32_t stride_a, uint32_t stride_b, uint32_t stride_d,
uint32_t batch_stride_a, uint32_t batch_stride_b, uint32_t batch_stride_d,
uint32_t split_k, uint32_t batch, uint32_t ne02, uint32_t ne12, uint32_t broadcast2, uint32_t broadcast3,
uint32_t split_k, uint32_t stream_k_groups, uint32_t batch, uint32_t ne02, uint32_t ne12, uint32_t broadcast2, uint32_t broadcast3,
uint32_t padded_n, uint64_t b_col_bytes, uint64_t d_col_bytes) {
VK_LOG_DEBUG("ggml_vk_matmul(a: (" << a.buffer->buffer << ", " << a.offset << ", " << a.size << "), b: (" << b.buffer->buffer << ", " << b.offset << ", " << b.size << "), d: (" << d.buffer->buffer << ", " << d.offset << ", " << d.size << "), split_k: (" << (split_k_buffer.buffer != nullptr ? split_k_buffer.buffer->buffer : VK_NULL_HANDLE) << ", " << split_k_buffer.offset << ", " << split_k_buffer.size << "), m: " << m << ", n: " << n << ", k: " << k << ", stride_a: " << stride_a << ", stride_b: " << stride_b << ", stride_d: " << stride_d << ", batch_stride_a: " << batch_stride_a << ", batch_stride_b: " << batch_stride_b << ", batch_stride_d: " << batch_stride_d << ", split_k: " << split_k << ", batch: " << batch << ", ne02: " << ne02 << ", ne12: " << ne12 << ", broadcast2: " << broadcast2 << ", broadcast3: " << broadcast3 << ", padded_n: " << padded_n << ")");
if (stream_k_groups != 0) {
GGML_ASSERT(batch_stride_d == m * n);
if (ctx->prealloc_split_k_need_sync) {
ggml_vk_sync_buffers(ctx, subctx);
}
ggml_pipeline_request_descriptor_sets(ctx, pipeline, 1);
ggml_pipeline_request_descriptor_sets(ctx, ctx->device->pipeline_matmul_stream_k_fixup, 1);
const uint32_t bm = pipeline->wg_denoms[0];
const uint32_t bn = pipeline->wg_denoms[1];
const uint32_t k_units_per_tile = k / 256;
const uint32_t blocks_m = m / bm;
const uint32_t blocks_n = n / bn;
const uint32_t total_work_units = blocks_m * blocks_n * batch * k_units_per_tile;
const uint32_t work_units_per_group = total_work_units / stream_k_groups;
const uint32_t groups_with_extra_unit = total_work_units % stream_k_groups;
vk_mat_mat_push_constants pc1 = {
m, n, k, stride_a, stride_b, stride_d, batch_stride_a, batch_stride_b, batch_stride_d, 0, batch, k, ne02, ne12, broadcast2, broadcast3, padded_n,
k_units_per_tile, work_units_per_group, groups_with_extra_unit,
0, 0, 0, 0, 0, 0, 0, 0
};
init_pushconst_fastdiv(pc1, blocks_m, blocks_n);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { a, b, d, split_k_buffer }, pc1, { stream_k_groups * pipeline->wg_denoms[0], 1, 1 });
ggml_vk_sync_buffers(ctx, subctx);
vk_mat_mat_stream_k_fixup_push_constants pc2 = {
m, n, bm, bn, k_units_per_tile, blocks_m, blocks_n, work_units_per_group, groups_with_extra_unit,
0, 0, 0, 0, 0, 0, 0
};
init_pushconst_fastdiv(pc2);
ggml_vk_dispatch_pipeline(ctx, subctx, ctx->device->pipeline_matmul_stream_k_fixup, { split_k_buffer, d }, pc2, { stream_k_groups, bm, 1 });
ctx->prealloc_split_k_need_sync = true;
return;
}
if (split_k == 1) {
// a small tile can still produce more workgroups than the device allows on the n axis:
// split the dispatch into n-chunks, shifting the B and D column windows
@@ -6438,8 +6603,12 @@ void ggml_vk_matmul(
while (base_work_group_z < batch) {
uint32_t groups_z = std::min(batch - base_work_group_z, ctx->device->properties.limits.maxComputeWorkGroupCount[2]);
const vk_mat_mat_push_constants pc = { m, n_chunk, k, stride_a, stride_b, stride_d, batch_stride_a, batch_stride_b, batch_stride_d, base_work_group_z, batch, k, ne02, ne12, broadcast2, broadcast3, uint32_t(padded_n - n_base) };
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { a, b_chunk, d_chunk }, pc, { m, n_chunk, groups_z });
vk_mat_mat_push_constants pc = {
m, n_chunk, k, stride_a, stride_b, stride_d, batch_stride_a, batch_stride_b, batch_stride_d, base_work_group_z, batch, k, ne02, ne12, broadcast2, broadcast3, uint32_t(padded_n - n_base),
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
};
init_pushconst_fastdiv(pc);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { a, b_chunk, d_chunk, d_chunk }, pc, { m, n_chunk, groups_z });
base_work_group_z += groups_z;
}
}
@@ -6462,9 +6631,13 @@ void ggml_vk_matmul(
while (base_work_group_z < batch) {
uint32_t groups_z = std::min(batch - base_work_group_z, ctx->device->properties.limits.maxComputeWorkGroupCount[2]);
const vk_mat_mat_push_constants pc1 = { m, n, k, stride_a, stride_b, stride_d, batch_stride_a, batch_stride_b, batch_stride_d, base_work_group_z, batch, k_split, ne02, ne12, broadcast2, broadcast3, padded_n };
vk_mat_mat_push_constants pc1 = {
m, n, k, stride_a, stride_b, stride_d, batch_stride_a, batch_stride_b, batch_stride_d, base_work_group_z, batch, k_split, ne02, ne12, broadcast2, broadcast3, padded_n,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
};
init_pushconst_fastdiv(pc1);
// Make sure enough workgroups get assigned for split k to work
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { a, b, split_k_buffer }, pc1, { (CEIL_DIV(m, pipeline->wg_denoms[0]) * pipeline->wg_denoms[0]) * split_k, n, groups_z });
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { a, b, split_k_buffer, split_k_buffer }, pc1, { (CEIL_DIV(m, pipeline->wg_denoms[0]) * pipeline->wg_denoms[0]) * split_k, n, groups_z });
base_work_group_z += groups_z;
}
ggml_vk_sync_buffers(ctx, subctx);
@@ -6518,10 +6691,11 @@ static bool ggml_vk_get_mul_mat_mat_f16acc(ggml_backend_vk_context * ctx, ggml_t
return ctx->device->fp16 && prec_allows_f16;
}
static const std::vector<vk_matmul_pipeline_pair>* ggml_vk_get_mul_mat_mat_pipeline_map(
ggml_backend_vk_context * ctx, ggml_type src0_type, ggml_type src1_type, ggml_prec prec, bool mul_mat_id = false) {
static const std::vector<vk_matmul_pipeline_pair> * ggml_vk_get_mul_mat_mat_pipeline_map(
ggml_backend_vk_context * ctx, ggml_type src0_type, ggml_type src1_type, ggml_prec prec,
bool mul_mat_id = false, bool stream_k = false) {
bool f16acc = ggml_vk_get_mul_mat_mat_f16acc(ctx, src0_type, src1_type, prec);
vk_matmul_pipeline_key key{src0_type, src1_type, mul_mat_id, f16acc};
vk_matmul_pipeline_key key{src0_type, src1_type, mul_mat_id, f16acc, stream_k};
auto it = ctx->device->pipeline_matmul.find(key);
if (it == ctx->device->pipeline_matmul.end() || it->second.empty()) {
// Try without f16acc
@@ -6894,25 +7068,50 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
const bool y_f32_kernel = y_kernel_type == GGML_TYPE_F32;
const bool y_needs_reformat = !quantize_y && (!ggml_vk_dim01_contiguous(src1) || y_kernel_type != src1->type);
const std::vector<vk_matmul_pipeline_pair>* mmp_map =
ggml_vk_get_mul_mat_mat_pipeline_map(ctx, src0->type, y_kernel_type, ggml_prec_get_acc(dst));
ggml_type mmp_type_a = src0->type;
const ggml_type mmp_type_b = y_kernel_type;
const std::vector<vk_matmul_pipeline_pair> * mmp_map =
ggml_vk_get_mul_mat_mat_pipeline_map(ctx, mmp_type_a, mmp_type_b, ggml_prec_get_acc(dst));
const bool qx_needs_dequant = mmp_map == nullptr || x_non_contig;
const bool qy_needs_dequant = y_needs_reformat;
if (qx_needs_dequant) {
// dequant src0 to f16
mmp_map = ggml_vk_get_mul_mat_mat_pipeline_map(ctx, f16_type, y_kernel_type, ggml_prec_get_acc(dst));
mmp_type_a = f16_type;
mmp_map = ggml_vk_get_mul_mat_mat_pipeline_map(ctx, mmp_type_a, mmp_type_b, ggml_prec_get_acc(dst));
}
GGML_ASSERT(mmp_map != nullptr);
const std::vector<vk_matmul_pipeline_pair> * regular_mmp_map = mmp_map;
bool use_stream_k = ctx->device->coopmat2 && !ctx->device->disable_stream_k && !disable_split_k;
if (use_stream_k) {
mmp_map = ggml_vk_get_mul_mat_mat_pipeline_map(ctx, mmp_type_a, mmp_type_b, ggml_prec_get_acc(dst), false, true);
if (mmp_map == nullptr) {
mmp_map = regular_mmp_map;
use_stream_k = false;
}
}
const uint32_t kpad = quantize_y ? 0 : ggml_vk_align_size(ne10, ggml_vk_guess_matmul_pipeline_align_map(ctx, *mmp_map, ne01, ne11, false));
const bool aligned = !quantize_y && ne10 == kpad && ne01 > 8 && ne11 > 8;
const bool needs_64b_indexing = ggml_nbytes(src0) > ctx->device->properties.limits.maxStorageBufferRange;
if (use_stream_k && !aligned) {
mmp_map = regular_mmp_map;
use_stream_k = false;
}
vk_pipeline pipeline = ggml_vk_guess_matmul_pipeline_map(ctx, *mmp_map, ne01, ne11, aligned, false);
if (ggml_nbytes(src0) > ctx->device->properties.limits.maxStorageBufferRange) {
const bool narrow_n = mmp_map->back().narrow_n_max != 0 && ne11 <= mmp_map->back().narrow_n_max;
const uint32_t split_k = ggml_vk_guess_split_k(ctx, ne01, ne11, ne10, disable_split_k, pipeline, narrow_n);
const uint32_t stream_k_groups = use_stream_k ? ggml_vk_guess_stream_k_groups(ctx, pipeline, ne01, ne11, ne10, stride_d, ne12 * ne13, split_k, disable_split_k) : 0;
if (use_stream_k && stream_k_groups == 0) {
pipeline = ggml_vk_guess_matmul_pipeline_map(ctx, *regular_mmp_map, ne01, ne11, aligned, false);
}
if (needs_64b_indexing) {
pipeline = ggml_vk_get_64b_indexing_pipeline(ctx, pipeline);
}
@@ -6923,9 +7122,6 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
const uint64_t y_ne = padded_n * ne10 * ne12 * ne13;
const uint64_t d_ne = ggml_nelements(dst);
const bool narrow_n = mmp_map->back().narrow_n_max != 0 && ne11 <= mmp_map->back().narrow_n_max;
const uint32_t split_k = ggml_vk_guess_split_k(ctx, ne01, ne11, ne10, disable_split_k, pipeline, narrow_n);
const uint64_t qx_sz = ggml_type_size(src0->type) * x_ne / ggml_blck_size(src0->type);
const uint64_t qy_sz = ggml_type_size(src1->type) * y_ne / ggml_blck_size(src1->type);
const uint64_t x_sz = !qx_needs_dequant ? qx_sz : sizeof(ggml_fp16_t) * x_ne;
@@ -6953,12 +7149,13 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
to_q8_1 = ggml_vk_get_quantize_pipeline(ctx, GGML_TYPE_Q8_1);
}
const uint64_t stream_k_size = stream_k_groups != 0 ? uint64_t(stream_k_groups) * pipeline->wg_denoms[0] * pipeline->wg_denoms[1] * sizeof(float) : 0;
const uint64_t split_k_size = stream_k_groups != 0 ? stream_k_size : (split_k > 1 ? d_sz * split_k : 0);
{
const uint64_t split_k_size = split_k > 1 ? d_sz * split_k : 0;
if (
(qx_needs_dequant && x_sz > ctx->device->properties.limits.maxStorageBufferRange) ||
(qy_needs_dequant && y_sz > ctx->device->properties.limits.maxStorageBufferRange) ||
(split_k > 1 && split_k_size > ctx->device->properties.limits.maxStorageBufferRange)) {
(split_k_size > ctx->device->properties.limits.maxStorageBufferRange)) {
GGML_ABORT("Requested preallocation size is too large");
}
if (qx_needs_dequant && ctx->prealloc_size_x < x_sz) {
@@ -6969,7 +7166,7 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
ctx->prealloc_size_y = y_sz;
ggml_vk_preallocate_buffers(ctx, subctx);
}
if (split_k > 1 && ctx->prealloc_size_split_k < split_k_size) {
if (split_k_size != 0 && ctx->prealloc_size_split_k < split_k_size) {
ctx->prealloc_size_split_k = split_k_size;
ggml_vk_preallocate_buffers(ctx, subctx);
}
@@ -6984,7 +7181,7 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
if (quantize_y) {
ggml_pipeline_request_descriptor_sets(ctx, to_q8_1, 1);
}
if (split_k > 1) {
if (stream_k_groups == 0 && split_k > 1) {
ggml_pipeline_request_descriptor_sets(ctx, ctx->device->pipeline_matmul_split_k_reduce, 1);
}
}
@@ -7092,10 +7289,10 @@ static void ggml_vk_mul_mat_q_f16(ggml_backend_vk_context * ctx, vk_context& sub
ggml_vk_matmul(
ctx, subctx, pipeline,
{ d_X, x_buf_offset, x_range }, { d_Y, y_buf_offset, y_range },
ggml_vk_subbuffer(ctx, d_D, d_buf_offset), { ctx->prealloc_split_k, 0, d_sz * split_k },
ggml_vk_subbuffer(ctx, d_D, d_buf_offset), { ctx->prealloc_split_k, 0, split_k_size },
ne01, ne11, ne10,
ne10, ne10, stride_d, stride_batch_x, stride_batch_y, stride_batch_d,
split_k, ne12*ne13, ne02, ne12, r2, r3, padded_n,
split_k, stream_k_groups, ne12*ne13, ne02, ne12, r2, r3, padded_n,
b_col_bytes, (uint64_t)dst->nb[1]
); // NOLINT
@@ -7234,7 +7431,7 @@ static void ggml_vk_mul_mat_vec_q_f16(ggml_backend_vk_context * ctx, vk_context&
const ggml_type y_kernel_type = widen_bf16 ? GGML_TYPE_F32 : src1->type;
const bool f16_f32_kernel = y_kernel_type == GGML_TYPE_F32;
bool quantize_y = ctx->device->integer_dot_product && src1->type == GGML_TYPE_F32 && ggml_is_contiguous(src1) && !y_non_contig && (ne11 * ne10) % 4 == 0 && (swap_inputs || ggml_vk_src1_prec_allows_q8(dst)) && ggml_vk_should_use_mmvq(ctx->device, ne01, ne11, ne10, src0->type);
bool quantize_y = ctx->device->integer_dot_signed && src1->type == GGML_TYPE_F32 && ggml_is_contiguous(src1) && !y_non_contig && (ne11 * ne10) % 4 == 0 && (swap_inputs || ggml_vk_src1_prec_allows_q8(dst)) && ggml_vk_should_use_mmvq(ctx->device, ne01, ne11, ne10, src0->type);
vk_pipeline to_fp16_vk_0 = nullptr;
vk_pipeline to_fp16_vk_1 = nullptr;
@@ -8302,7 +8499,7 @@ static void ggml_vk_mul_mat_vec_id_q_f16(ggml_backend_vk_context * ctx, vk_conte
const bool y_non_contig = !ggml_vk_dim01_contiguous(src1);
const bool f16_f32_kernel = src1->type == GGML_TYPE_F32;
bool quantize_y = ctx->device->integer_dot_product && src1->type == GGML_TYPE_F32 && ggml_is_contiguous(src1) && !y_non_contig && (ne11 * ne10) % 4 == 0 && ggml_vk_src1_prec_allows_q8(dst) && ggml_vk_should_use_mmvq(ctx->device, ne01, ne12, ne10, src0->type);
bool quantize_y = ctx->device->integer_dot_signed && src1->type == GGML_TYPE_F32 && ggml_is_contiguous(src1) && !y_non_contig && (ne11 * ne10) % 4 == 0 && ggml_vk_src1_prec_allows_q8(dst) && ggml_vk_should_use_mmvq(ctx->device, ne01, ne12, ne10, src0->type);
vk_pipeline to_fp16_vk_0 = nullptr;
vk_pipeline to_fp16_vk_1 = nullptr;
@@ -10055,6 +10252,8 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const
return ctx->device->pipeline_arange_f32;
}
return nullptr;
case GGML_OP_SLEEP:
return ctx->device->pipeline_sleep;
case GGML_OP_FILL:
if (dst->type == GGML_TYPE_F32) {
return ctx->device->pipeline_fill_f32;
@@ -11291,6 +11490,31 @@ void ggml_vk_fill(ggml_backend_vk_context * ctx, vk_context& subctx, ggml_tensor
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { dst_buf }, pc, elements);
}
static void ggml_vk_sleep(ggml_backend_vk_context * ctx, vk_context& subctx, ggml_tensor * dst) {
VK_LOG_DEBUG("ggml_vk_sleep(dst=" << dst << ", us=" << ggml_get_op_params_i32(dst, 0) << ")");
// Vulkan does not specify the period of the shader realtime clock this assumes it matches the timestamp counter.
// A driver where this does not hold could sleep too short or too long by that ratio
const float ns_per_tick = ctx->device->properties.limits.timestampPeriod;
const uint64_t ns = 1000 * (uint64_t) ggml_get_op_params_i32(dst, 0);
const uint64_t ticks = std::min<uint64_t>(ns_per_tick > 0.0f ? (uint64_t)(ns / ns_per_tick) : ns, std::numeric_limits<uint32_t>::max());
vk_op_sleep_push_constants pc = {
(uint32_t) ticks,
};
vk_pipeline pipeline = ggml_vk_op_get_pipeline(ctx, nullptr, nullptr, nullptr, dst, GGML_OP_SLEEP);
GGML_ASSERT(pipeline != nullptr);
ggml_pipeline_request_descriptor_sets(ctx, pipeline, 1);
vk_subbuffer dst_buf = ggml_vk_tensor_subbuffer(ctx, dst);
// a single invocation spins, the buffer is bound only to keep the spin from being optimized out
std::array<uint32_t, 3> elements = { 1, 1, 1 };
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { dst_buf }, pc, elements);
}
void ggml_vk_sin(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, ggml_tensor * dst) {
ggml_vk_op_f32(ctx, subctx, src0, nullptr, nullptr, nullptr, dst, GGML_OP_SIN, vk_op_unary_push_constants_init(src0, dst));
}
@@ -13137,6 +13361,10 @@ bool ggml_vk_build_graph(ggml_backend_vk_context * ctx, ggml_cgraph * cgraph, in
case GGML_OP_FILL:
ggml_vk_fill(ctx, compute_ctx, node);
break;
case GGML_OP_SLEEP:
ggml_vk_sleep(ctx, compute_ctx, node);
break;
case GGML_OP_SCALE:
if (ctx->fused_hc_post_gate) {
@@ -16588,6 +16816,8 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
return op->type == GGML_TYPE_F32;
case GGML_OP_FILL:
return op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16;
case GGML_OP_SLEEP:
return device->shader_clock;
case GGML_OP_SCALE:
return ggml_is_contiguous(op->src[0]) && ggml_vk_is_float_elementwise(op);
case GGML_OP_PAD:
@@ -22,6 +22,6 @@ void main() {
if (i10 == i11) {
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(data_a[get_aoffset() + i13*p.nb03 + i12*p.nb02 + 0*p.nb01 + i10*p.nb00]);
} else {
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(0);
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(0.0);
}
}
@@ -0,0 +1,86 @@
#version 450
#include "utils.glsl"
layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;
layout (binding = 0) readonly buffer S4 {vec4 data_s4[];};
layout (binding = 1) buffer D4 {vec4 data_d4[];};
layout (push_constant) uniform parameter {
uint M;
uint N;
uint BM;
uint BN;
uint k_units_per_tile;
uint blocks_m;
uint blocks_n;
uint work_units_per_group;
uint groups_with_extra_unit;
uint k_units_mp;
uint blocks_m_mp;
uint blocks_n_mp;
uint work_units_mp;
uint large_group_work_mp;
uint div_Ls0;
uint div_Ls1;
} p;
const uint RESOLVE_ROWS = 32;
// A normal matmul workgroup accumulates one BM x BN output tile over the full K range.
// Stream-K divides that accumulation into work units. Each work unit covers 256 consecutive K values and produces a BM x BN partial sum.
// Work units are ordered [K unit, M tile, N tile, batch] and divided into one contiguous range per main workgroup.
// A main workgroup writes its partial sum to D when it reaches the end of an output tile, or to scratch when its range stops within a tile.
// Scratch has shape [BM, BN, stream_k_groups] and stores float partial sums.
// Element [r, c, g] is workgroup g's contribution to local output element [r, c] in the last output tile of its range.
// The output tile for each scratch slot is derived from the workgroup's range and is not stored in scratch.
// Only the last output tile in a workgroup's range can stop early, so one scratch slot is enough.
// Fixup workgroup (x, y) handles main workgroup x and rows 32*y through 32*y+31.
// If main workgroup x starts within a tile and completes it, fixup adds the earlier groups' partial sums from scratch to D.
// Find the main workgroup whose range contains work.
// With q = work_units_per_group and r = groups_with_extra_unit, the first r workgroups get q + 1 units and all others get q.
// group(work) = work / (q + 1) when work < r*(q + 1), otherwise r + (work - r*(q + 1))/q.
uint group_for_work(uint work) {
const uint large_group_work = p.work_units_per_group + 1;
const uint large_work = large_group_work * p.groups_with_extra_unit;
if (work < large_work) {
return fastdiv(work, p.large_group_work_mp, fastdiv_L(p.div_Ls1, 1));
}
return p.groups_with_extra_unit + fastdiv(work - large_work, p.work_units_mp, fastdiv_L(p.div_Ls1, 0));
}
void main() {
const uint group_idx = gl_WorkGroupID.x;
const uint work_unit = group_idx * p.work_units_per_group + min(group_idx, p.groups_with_extra_unit);
const uint work_unit_end = work_unit + p.work_units_per_group + uint(group_idx < p.groups_with_extra_unit);
const uint tile_idx = fastdiv(work_unit, p.k_units_mp, fastdiv_L(p.div_Ls0, 0));
const uint tile_work = tile_idx * p.k_units_per_tile;
// Only a range that starts within a tile and completes it needs fixup.
if (work_unit == tile_work || work_unit_end < tile_work + p.k_units_per_tile) {
return;
}
const uint first_group = group_for_work(tile_work);
const uint outer_tile = fastdiv(tile_idx, p.blocks_m_mp, fastdiv_L(p.div_Ls0, 1));
const uint ir = tile_idx - outer_tile * p.blocks_m;
const uint batch_idx = fastdiv(outer_tile, p.blocks_n_mp, fastdiv_L(p.div_Ls0, 2));
const uint ic = outer_tile - batch_idx * p.blocks_n;
const uint row_base = gl_WorkGroupID.y * RESOLVE_ROWS;
for (uint stripe_elem = gl_LocalInvocationIndex * 4; stripe_elem < RESOLVE_ROWS * p.BN; stripe_elem += gl_WorkGroupSize.x * 4) {
const uint r = row_base + (stripe_elem & (RESOLVE_ROWS - 1));
const uint c = stripe_elem / RESOLVE_ROWS;
const uint elem_idx = c * p.BM + r;
const uint dst_idx = batch_idx * p.M * p.N + (ic * p.BN + c) * p.M + ir * p.BM + r;
// D contains the completing segment. Add the earlier segments from scratch.
vec4 result = data_d4[dst_idx / 4];
for (uint previous_group = first_group; previous_group < group_idx; ++previous_group) {
result += data_s4[(previous_group * p.BM * p.BN + elem_idx) / 4];
}
data_d4[dst_idx / 4] = result;
}
}
+137 -44
View File
@@ -38,6 +38,7 @@ layout (constant_id = 12) const uint MmTypeA = 0;
#include "types.glsl"
#include "dot_product_funcs.glsl"
#include "utils.glsl"
#ifndef MULMAT_QUANT
#ifndef LOAD_VEC_A
@@ -121,6 +122,9 @@ layout (binding = 0) readonly buffer BUF_Q5_K_P32 { block_q5_K_packed32 data[]
layout (binding = 1) readonly buffer B {B_TYPE data_b[];};
layout (binding = 1) readonly buffer B_SCALAR {B_TYPE_SCALAR data_b_scalar[];};
layout (binding = 2) writeonly buffer D {D_TYPE data_d[];};
#ifndef MUL_MAT_ID
layout (binding = 3) writeonly buffer S {D_TYPE data_s[];};
#endif
#ifdef MUL_MAT_ID
layout (binding = 3) readonly buffer IDS {int data_ids[];};
@@ -155,6 +159,18 @@ layout (push_constant) uniform parameter
uint ne12;
uint broadcast2;
uint broadcast3;
uint padded_N;
uint k_units_per_tile;
uint work_units_per_group;
uint groups_with_extra_unit;
uint k_units_mp;
uint blocks_m_mp;
uint blocks_n_mp;
uint ne12_mp;
uint broadcast2_mp;
uint broadcast3_mp;
uint div_Ls0;
uint div_Ls1;
#endif
} p;
@@ -189,6 +205,9 @@ layout(constant_id = 13) const bool APPLY_SLM_A_RESHAPE = false;
const uint SHMEM_STRIDE_PAD = 1;
const bool APPLY_SLM_A_RESHAPE = false;
#endif
#ifndef MUL_MAT_ID
layout(constant_id = 15) const bool stream_k = false;
#endif
#define SHMEM_STRIDE (BK / 2 + SHMEM_STRIDE_PAD)
shared FLOAT_TYPEV2 buf_a[BM * SHMEM_STRIDE];
@@ -218,27 +237,24 @@ void main() {
_ne1 = data_row_map[2 + 2 * tile_idx];
const uint ic = 0;
#else
const uint ic = gl_WorkGroupID.y;
const uint group_idx = gl_WorkGroupID.x;
uint stream_k_work_unit;
uint stream_k_work_unit_end;
if (stream_k) {
stream_k_work_unit = group_idx * p.work_units_per_group + min(group_idx, p.groups_with_extra_unit);
stream_k_work_unit_end = stream_k_work_unit + p.work_units_per_group + uint(group_idx < p.groups_with_extra_unit);
}
#endif
#if defined(NEEDS_INIT_IQ_SHMEM) || defined(MULMAT_QUANT)
init_iq_shmem(gl_WorkGroupSize);
#endif
#ifndef MUL_MAT_ID
const uint batch_idx = gl_WorkGroupID.z + p.base_work_group_z;
const uint i13 = batch_idx / p.ne12;
const uint i12 = batch_idx % p.ne12;
const uint i03 = i13 / p.broadcast3;
const uint i02 = i12 / p.broadcast2;
const uint batch_idx_a = i03 * p.ne02 + i02;
#endif
#ifdef MUL_MAT_ID
const uint blocks_m = (p.M + BM - 1) / BM;
const uint ir = gl_WorkGroupID.x % blocks_m;
const uint ik = gl_WorkGroupID.x / blocks_m;
#endif
const uint WNITER = (WM * WN) / (WARP * TM * TN * WMITER);
const uint WSUBM = WM / WMITER;
@@ -288,6 +304,41 @@ void main() {
const uint loadstride_a = gl_WorkGroupSize.x * LOAD_VEC_A_EFF * LOAD_VEC_BATCH_A / BK;
const uint loadstride_b = gl_WorkGroupSize.x * LOAD_VEC_B_EFF * LOAD_VEC_BATCH_B / BK;
#ifndef MUL_MAT_ID
do {
uint stream_k_tile_k_unit;
uint stream_k_tile_k_units;
uint ir;
uint ic;
uint batch_idx;
uint ik;
if (stream_k) {
const uint tile_idx = fastdiv(stream_k_work_unit, p.k_units_mp, fastdiv_L(p.div_Ls0, 0));
stream_k_tile_k_unit = stream_k_work_unit - tile_idx * p.k_units_per_tile;
stream_k_tile_k_units = min(stream_k_work_unit_end - stream_k_work_unit, p.k_units_per_tile - stream_k_tile_k_unit);
const uint blocks_m = p.M / BM;
const uint blocks_n = p.N / BN;
const uint outer_tile = fastdiv(tile_idx, p.blocks_m_mp, fastdiv_L(p.div_Ls0, 1));
ir = tile_idx - outer_tile * blocks_m;
batch_idx = fastdiv(outer_tile, p.blocks_n_mp, fastdiv_L(p.div_Ls0, 2));
ic = outer_tile - batch_idx * blocks_n;
} else {
const uint blocks_m = (p.M + BM - 1) / BM;
ic = gl_WorkGroupID.y;
batch_idx = gl_WorkGroupID.z + p.base_work_group_z;
ir = group_idx % blocks_m;
ik = group_idx / blocks_m;
}
const uint i13 = fastdiv(batch_idx, p.ne12_mp, fastdiv_L(p.div_Ls1, 0));
const uint i12 = batch_idx - i13 * p.ne12;
const uint i03 = fastdiv(i13, p.broadcast3_mp, fastdiv_L(p.div_Ls1, 2));
const uint i02 = fastdiv(i12, p.broadcast2_mp, fastdiv_L(p.div_Ls1, 1));
const uint batch_idx_a = i03 * p.ne02 + i02;
#endif
#ifdef MUL_MAT_ID
load_row_ids(row_begin);
uint required_work_items = _ne1 * BK / LOAD_VEC_B_EFF / LOAD_VEC_BATCH_B;
@@ -298,8 +349,15 @@ void main() {
const uint start_k = 0;
const uint end_k = p.K;
#else
const uint start_k = ik * p.k_split;
const uint end_k = min(p.K, (ik + 1) * p.k_split);
uint start_k;
uint end_k;
if (stream_k) {
start_k = stream_k_tile_k_unit * 256;
end_k = (stream_k_tile_k_unit + stream_k_tile_k_units) * 256;
} else {
start_k = ik * p.k_split;
end_k = min(p.K, (ik + 1) * p.k_split);
}
#endif
uint pos_a =
@@ -438,7 +496,17 @@ void main() {
const uint dc = ic * BN + warp_c * WN;
#ifndef MUL_MAT_ID
const uint offsets = batch_idx * p.batch_stride_d + ik * p.batch_stride_d * p.num_batches;
uint offsets = batch_idx * p.batch_stride_d;
if (!stream_k) {
offsets += ik * p.batch_stride_d * p.num_batches;
}
#endif
#ifndef MUL_MAT_ID
bool stream_k_store_scratch;
if (stream_k) {
stream_k_store_scratch = stream_k_tile_k_unit + stream_k_tile_k_units != p.k_units_per_tile;
}
#endif
#ifdef COOPMAT
@@ -462,36 +530,44 @@ void main() {
}
}
#else
const bool is_aligned = p.stride_d % 4 == 0; // Assumption: D_TYPE == float
[[unroll]] for (uint cm_row = 0; cm_row < cms_per_row; cm_row++) {
[[unroll]] for (uint cm_col = 0; cm_col < cms_per_col; cm_col++) {
const bool is_in_bounds = dr + (cm_row + 1) * TM <= p.M && dc + (cm_col + 1) * TN <= p.N;
if (is_aligned && is_in_bounds) {
// Full coopMat is within bounds and stride_d is aligned with 16B
coopmat<D_TYPE, gl_ScopeSubgroup, TM, TN, gl_MatrixUseAccumulator> cm_dtype = coopmat<D_TYPE, gl_ScopeSubgroup, TM, TN, gl_MatrixUseAccumulator>(sums[cm_col * cms_per_row + cm_row]);
coopMatStore(cm_dtype, data_d, offsets + (dc + cm_col * TN) * p.stride_d + dr + cm_row * TM, p.stride_d, gl_CooperativeMatrixLayoutColumnMajor);
} else if (is_in_bounds) {
// Full coopMat is within bounds, but stride_d is not aligned
coopMatStore(sums[cm_col * cms_per_row + cm_row], coopmat_stage, warp_i * TM * TN, TM, gl_CooperativeMatrixLayoutColumnMajor);
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
[[unroll]] for (uint col = 0; col < TN; col += storestride) {
data_d[offsets + (dc + cm_col * TN + col + store_c) * p.stride_d + dr + cm_row * TM + store_r] = D_TYPE(coopmat_stage[warp_i * TM * TN + (col + store_c) * TM + store_r]);
coopmat<D_TYPE, gl_ScopeSubgroup, TM, TN, gl_MatrixUseAccumulator> cm_dtype = coopmat<D_TYPE, gl_ScopeSubgroup, TM, TN, gl_MatrixUseAccumulator>(sums[cm_col * cms_per_row + cm_row]);
if (stream_k) {
if (stream_k_store_scratch) {
const uint scratch_offset = group_idx * BM * BN + (warp_c * WN + cm_col * TN) * BM + warp_r * WM + cm_row * TM;
coopMatStore(cm_dtype, data_s, scratch_offset, BM, gl_CooperativeMatrixLayoutColumnMajor);
} else {
coopMatStore(cm_dtype, data_d, offsets + (dc + cm_col * TN) * p.stride_d + dr + cm_row * TM, p.stride_d, gl_CooperativeMatrixLayoutColumnMajor);
}
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
} else if (dr + cm_row * TM < p.M && dc + cm_col * TN < p.N) {
// Partial coopMat is within bounds
coopMatStore(sums[cm_col * cms_per_row + cm_row], coopmat_stage, warp_i * TM * TN, TM, gl_CooperativeMatrixLayoutColumnMajor);
} else {
const bool is_aligned = p.stride_d % 4 == 0; // Assumption: D_TYPE == float
const bool is_in_bounds = dr + (cm_row + 1) * TM <= p.M && dc + (cm_col + 1) * TN <= p.N;
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
[[unroll]] for (uint col = 0; col < TN; col += storestride) {
if (dr + cm_row * TM + store_r < p.M && dc + cm_col * TN + col + store_c < p.N) {
if (is_aligned && is_in_bounds) {
// Full coopMat is within bounds and stride_d is aligned with 16B
coopMatStore(cm_dtype, data_d, offsets + (dc + cm_col * TN) * p.stride_d + dr + cm_row * TM, p.stride_d, gl_CooperativeMatrixLayoutColumnMajor);
} else if (is_in_bounds) {
// Full coopMat is within bounds, but stride_d is not aligned
coopMatStore(sums[cm_col * cms_per_row + cm_row], coopmat_stage, warp_i * TM * TN, TM, gl_CooperativeMatrixLayoutColumnMajor);
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
[[unroll]] for (uint col = 0; col < TN; col += storestride) {
data_d[offsets + (dc + cm_col * TN + col + store_c) * p.stride_d + dr + cm_row * TM + store_r] = D_TYPE(coopmat_stage[warp_i * TM * TN + (col + store_c) * TM + store_r]);
}
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
} else if (dr + cm_row * TM < p.M && dc + cm_col * TN < p.N) {
// Partial coopMat is within bounds
coopMatStore(sums[cm_col * cms_per_row + cm_row], coopmat_stage, warp_i * TM * TN, TM, gl_CooperativeMatrixLayoutColumnMajor);
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
[[unroll]] for (uint col = 0; col < TN; col += storestride) {
if (dr + cm_row * TM + store_r < p.M && dc + cm_col * TN + col + store_c < p.N) {
data_d[offsets + (dc + cm_col * TN + col + store_c) * p.stride_d + dr + cm_row * TM + store_r] = D_TYPE(coopmat_stage[warp_i * TM * TN + (col + store_c) * TM + store_r]);
}
}
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
}
controlBarrier(gl_ScopeSubgroup, gl_ScopeSubgroup, gl_StorageSemanticsShared, gl_SemanticsAcquireRelease);
}
}
}
@@ -519,11 +595,22 @@ void main() {
data_d[row_idx.y * p.batch_stride_d + row_idx.x * p.stride_d + dr_warp + 2 * cr + 1] = D_TYPE(sums[sums_idx].y);
}
#else
if (dr_warp + 2 * cr < p.M && dc_warp + cc < p.N) {
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr] = D_TYPE(sums[sums_idx].x);
}
if (dr_warp + 2 * cr + 1 < p.M && dc_warp + cc < p.N) {
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr + 1] = D_TYPE(sums[sums_idx].y);
if (stream_k) {
const uint scratch_offset = group_idx * BM * BN + (dc_warp + cc - ic * BN) * BM + dr_warp + 2 * cr - ir * BM;
if (stream_k_store_scratch) {
data_s[scratch_offset] = D_TYPE(sums[sums_idx].x);
data_s[scratch_offset + 1] = D_TYPE(sums[sums_idx].y);
} else {
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr] = D_TYPE(sums[sums_idx].x);
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr + 1] = D_TYPE(sums[sums_idx].y);
}
} else {
if (dr_warp + 2 * cr < p.M && dc_warp + cc < p.N) {
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr] = D_TYPE(sums[sums_idx].x);
}
if (dr_warp + 2 * cr + 1 < p.M && dc_warp + cc < p.N) {
data_d[offsets + (dc_warp + cc) * p.stride_d + dr_warp + 2 * cr + 1] = D_TYPE(sums[sums_idx].y);
}
}
#endif // MUL_MAT_ID
}
@@ -531,4 +618,10 @@ void main() {
}
}
#endif // COOPMAT
#ifndef MUL_MAT_ID
if (stream_k) {
stream_k_work_unit += stream_k_tile_k_units;
}
} while (stream_k && stream_k_work_unit < stream_k_work_unit_end);
#endif
}
@@ -61,6 +61,10 @@ uint mm_quant_k() {
}
#endif
#ifndef MUL_MAT_ID
layout (constant_id = 15) const bool stream_k = false;
#endif
layout (push_constant) uniform parameter
{
uint M;
@@ -91,6 +95,17 @@ layout (push_constant) uniform parameter
uint broadcast3;
// N dimension for the B matrix can be >= p.N
uint padded_N;
uint k_units_per_tile;
uint work_units_per_group;
uint groups_with_extra_unit;
uint k_units_mp;
uint blocks_m_mp;
uint blocks_n_mp;
uint ne12_mp;
uint broadcast2_mp;
uint broadcast3_mp;
uint div_Ls0;
uint div_Ls1;
#endif
} p;
@@ -102,6 +117,9 @@ layout (binding = 0) readonly buffer A {uint8_t data_a[];};
#endif
layout (binding = 1) readonly buffer B {B_TYPE data_b[];};
layout (binding = 2) writeonly buffer D {D_TYPE data_d[];};
#ifndef MUL_MAT_ID
layout (binding = 3) writeonly buffer S {D_TYPE data_s[];};
#endif
#if defined(MUL_MAT_ID) && defined(GGML_VULKAN_COOPMAT2_DECODE_VECTOR)
layout (binding = 1) readonly buffer B4 {B_TYPEV4 data_b_v4[];};
#endif
@@ -261,6 +279,17 @@ void main() {
const uint row_begin = tile_idx * BN;
_ne1 = data_row_map[2 + 2 * tile_idx];
const uint ic = 0;
#else
const uint group_idx = gl_WorkGroupID.x;
uint stream_k_work_unit;
uint stream_k_work_unit_end;
if (stream_k) {
// A work unit covers 256 consecutive K elements of one output tile.
// Work units are flattened as [K unit, M tile, N tile, batch].
stream_k_work_unit = group_idx * p.work_units_per_group + min(group_idx, p.groups_with_extra_unit);
stream_k_work_unit_end = stream_k_work_unit + p.work_units_per_group + uint(group_idx < p.groups_with_extra_unit);
}
#endif
#ifdef MUL_MAT_ID
@@ -278,43 +307,61 @@ void main() {
#endif
#ifndef MUL_MAT_ID
const uint batch_idx = gl_WorkGroupID.z + p.base_work_group_z;
do {
uint stream_k_tile_k_unit;
uint stream_k_tile_k_units;
uint ir;
uint ic;
uint batch_idx;
uint ik;
const uint i13 = batch_idx / p.ne12;
const uint i12 = batch_idx % p.ne12;
if (stream_k) {
// The 256-element granularity does not split any supported quant block.
const uint tile_idx = fastdiv(stream_k_work_unit, p.k_units_mp, fastdiv_L(p.div_Ls0, 0));
stream_k_tile_k_unit = stream_k_work_unit - tile_idx * p.k_units_per_tile;
stream_k_tile_k_units = min(stream_k_work_unit_end - stream_k_work_unit, p.k_units_per_tile - stream_k_tile_k_unit);
const uint i03 = i13 / p.broadcast3;
const uint i02 = i12 / p.broadcast2;
const uint batch_idx_a = i03 * p.ne02 + i02;
const uint blocks_m = p.M / BM;
const uint blocks_n = p.N / BN;
const uint outer_tile = fastdiv(tile_idx, p.blocks_m_mp, fastdiv_L(p.div_Ls0, 1));
ir = tile_idx - outer_tile * blocks_m;
batch_idx = fastdiv(outer_tile, p.blocks_n_mp, fastdiv_L(p.div_Ls0, 2));
ic = outer_tile - batch_idx * blocks_n;
} else {
const uint blocks_m = (p.M + BM - 1) / BM;
// L2-friendly workgroup scheduling
const uint blocks_n = (p.N + BN - 1) / BN;
#if defined(MULMAT_QUANT)
const uint a_panel_bytes = (BM * p.K / mm_quant_k()) * MmABlockBytes;
#elif defined(DATA_A_F32)
const uint a_panel_bytes = BM * p.K * 4;
#elif QUANT_K == 1
const uint a_panel_bytes = BM * p.K * 2;
#else
const uint a_panel_bytes = (BM * p.K) / 2 + (BM * p.K) / 16;
#endif
const uint group_m = clamp(GROUP_A_BUDGET / max(a_panel_bytes, 1u), 1u, min(blocks_m, 32u));
const uint tiles_per_group = group_m * blocks_n;
const uint lin = gl_WorkGroupID.y * blocks_m + (gl_WorkGroupID.x % blocks_m);
const uint group_id = lin / tiles_per_group;
const uint first_m = group_id * group_m;
const uint gsize = min(blocks_m - first_m, group_m);
const uint in_group = lin - group_id * tiles_per_group;
ir = first_m + in_group % gsize;
ic = in_group / gsize;
batch_idx = gl_WorkGroupID.z + p.base_work_group_z;
ik = group_idx / blocks_m;
}
const uint i13 = fastdiv(batch_idx, p.ne12_mp, fastdiv_L(p.div_Ls1, 0));
const uint i12 = batch_idx - i13 * p.ne12;
const uint i03 = fastdiv(i13, p.broadcast3_mp, fastdiv_L(p.div_Ls1, 2));
const uint i02 = fastdiv(i12, p.broadcast2_mp, fastdiv_L(p.div_Ls1, 1));
const uint batch_idx_a = i03 * p.ne02 + i02;
#else
const uint blocks_m = (p.M + BM - 1) / BM;
const uint ik = gl_WorkGroupID.x / blocks_m;
#ifdef MUL_MAT_ID
const uint ir = gl_WorkGroupID.x % blocks_m;
#else
// L2-friendly workgroup scheduling
const uint blocks_n = (p.N + BN - 1) / BN;
#if defined(MULMAT_QUANT)
const uint a_panel_bytes = (BM * p.K / mm_quant_k()) * MmABlockBytes;
#elif defined(DATA_A_F32)
const uint a_panel_bytes = BM * p.K * 4;
#elif QUANT_K == 1
const uint a_panel_bytes = BM * p.K * 2;
#else
const uint a_panel_bytes = (BM * p.K) / 2 + (BM * p.K) / 16;
#endif
const uint group_m = clamp(GROUP_A_BUDGET / max(a_panel_bytes, 1u), 1u, min(blocks_m, 32u));
const uint tiles_per_group = group_m * blocks_n;
const uint lin = gl_WorkGroupID.y * blocks_m + (gl_WorkGroupID.x % blocks_m);
const uint group_id = lin / tiles_per_group;
const uint first_m = group_id * group_m;
const uint gsize = min(blocks_m - first_m, group_m);
const uint in_group = lin - group_id * tiles_per_group;
const uint ir = first_m + in_group % gsize;
const uint ic = in_group / gsize;
#endif
#ifdef MUL_MAT_ID
@@ -325,8 +372,15 @@ void main() {
uint start_k = 0;
const uint end_k = p.K;
#else
uint start_k = ik * p.k_split;
const uint end_k = min(p.K, (ik + 1) * p.k_split);
uint start_k;
uint end_k;
if (stream_k) {
start_k = stream_k_tile_k_unit * 256;
end_k = (stream_k_tile_k_unit + stream_k_tile_k_units) * 256;
} else {
start_k = ik * p.k_split;
end_k = min(p.K, (ik + 1) * p.k_split);
}
#endif
#ifdef MULMAT_QUANT
@@ -341,7 +395,10 @@ void main() {
#else
uint pos_a = batch_idx_a * (p.batch_stride_a / qk);
uint pos_b = batch_idx * p.batch_stride_b;
uint pos_d = batch_idx * p.batch_stride_d + ik * p.batch_stride_d * p.num_batches;
uint pos_d = batch_idx * p.batch_stride_d;
if (!stream_k) {
pos_d += ik * p.batch_stride_d * p.num_batches;
}
#endif
#ifdef MULMAT_QUANT
@@ -404,12 +461,15 @@ void main() {
const uint UNROLL_K = (qk == 256) ? 256 : (BK * 8);
const uint unroll_count = UNROLL_K / BK;
// Detect a fast path where all loads are entirely in bounds and no clamping is required
if ((ir + 1) * BM <= p.M && (ic + 1) * BN <= p.padded_N && (start_k % START_ALIGN_K) == 0 && (end_k % BK) == 0 &&
bool fast_path = stream_k;
if (!stream_k) {
fast_path = (ir + 1) * BM <= p.M && (ic + 1) * BN <= p.padded_N && (start_k % START_ALIGN_K) == 0 && (end_k % BK) == 0 &&
#if !defined(MULMAT_QUANT) && QUANT_K == 1
(stride_a % 8) == 0 &&
(stride_a % 8) == 0 &&
#endif
(stride_b % 8) == 0) {
(stride_b % 8) == 0;
}
if (fast_path) {
// Hint to the compiler that values are aligned (want 16B alignment)
start_k &= ~(START_ALIGN_K-1);
stride_b &= ~7;
@@ -427,7 +487,7 @@ void main() {
// The fetches and stores are pipelined to hide the latency.
fetch_scales(ir * BM, pos_a, stride_a, start_k, tid, true);
if (enable_smaller_matrices && ic * BN + BNover4 >= p.N) {
if (!stream_k && enable_smaller_matrices && ic * BN + BNover4 >= p.N) {
coopmat<ACC_TYPE, gl_ScopeWorkgroup, BM, BNover4, gl_MatrixUseAccumulator> sum = coopmat<ACC_TYPE, gl_ScopeWorkgroup, BM, BNover4, gl_MatrixUseAccumulator>(0.0);
for (uint i = 0; i < k_iters; ++i) {
@@ -470,7 +530,7 @@ void main() {
coopMatStoreTensorNV(mat_d, data_d, pos_d, sliceTensorLayoutNV(tensorLayoutD, ic * BN, BNover4, ir * BM, BM), tensorViewTranspose);
return;
} else if (enable_smaller_matrices && ic * BN + BNover2 >= p.N) {
} else if (!stream_k && enable_smaller_matrices && ic * BN + BNover2 >= p.N) {
coopmat<ACC_TYPE, gl_ScopeWorkgroup, BM, BNover2, gl_MatrixUseAccumulator> sum = coopmat<ACC_TYPE, gl_ScopeWorkgroup, BM, BNover2, gl_MatrixUseAccumulator>(0.0);
for (uint i = 0; i < k_iters; ++i) {
@@ -555,8 +615,18 @@ void main() {
coopmat<D_TYPE, gl_ScopeWorkgroup, BM, BN, gl_MatrixUseAccumulator> mat_d = coopmat<D_TYPE, gl_ScopeWorkgroup, BM, BN, gl_MatrixUseAccumulator>(sum);
coopMatStoreTensorNV(mat_d, data_d, pos_d, sliceTensorLayoutNV(tensorLayoutD, ic * BN, BN, ir * BM, BM), tensorViewTranspose);
return;
if (stream_k) {
if (stream_k_tile_k_unit + stream_k_tile_k_units != p.k_units_per_tile) {
tensorLayoutNV<2> tensorLayoutS = createTensorLayoutNV(2);
tensorLayoutS = setTensorLayoutDimensionNV(tensorLayoutS, BN, BM);
tensorLayoutS = setTensorLayoutStrideNV(tensorLayoutS, BM, 1);
coopMatStoreTensorNV(mat_d, data_s, group_idx * BM * BN, tensorLayoutS, tensorViewTranspose);
} else {
coopMatStoreTensorNV(mat_d, data_d, pos_d, sliceTensorLayoutNV(tensorLayoutD, ic * BN, BN, ir * BM, BM), tensorViewTranspose);
}
} else {
coopMatStoreTensorNV(mat_d, data_d, pos_d, sliceTensorLayoutNV(tensorLayoutD, ic * BN, BN, ir * BM, BM), tensorViewTranspose);
}
}
} else
#endif // !defined(MUL_MAT_ID)
@@ -720,4 +790,10 @@ void main() {
coopMatStoreTensorNV(mat_d, data_d, pos_d, sliceTensorLayoutNV(tensorLayoutD, ic * BN, BN, ir * BM, BM), tensorViewTranspose);
#endif
}
#ifndef MUL_MAT_ID
if (stream_k) {
stream_k_work_unit += stream_k_tile_k_units;
}
} while (stream_k && stream_k_work_unit < stream_k_work_unit_end);
#endif
}
@@ -82,6 +82,8 @@ layout (constant_id = 7) const uint TM = 4;
layout (constant_id = 8) const uint TN = 2;
layout (constant_id = 9) const uint TK = 1; // Only needed for coopmat
layout (constant_id = 10) const uint WARP = 32;
// Use the mixed signedness dot product, for devices that accelerate only that one
layout (constant_id = 11) const bool MMQ_SUDOT = false;
#define BK 32
@@ -102,6 +104,13 @@ shared block_b_cache buf_b[BN * BK_STEP];
block_a_cache cache_a[WMITER * TM];
block_b_cache cache_b;
#if defined(DATA_A_Q8_0) || defined(DATA_A_MXFP4) || defined(DATA_A_IQ4_XS) || defined(DATA_A_IQ3_S) || defined(DATA_A_Q3_K) || defined(DATA_A_Q6_K)
// These quants are signed. MMQ_SUDOT gives them a bias and uses the sum of each half of the q8_1 block to remove it.
#define MMQ_B_SUMS
shared int32_t buf_b_qsum[MMQ_SUDOT ? BN * BK_STEP * 2 : 1];
int32_t cache_b_qsum[2];
#endif
#define LOAD_VEC_A (4 * QUANT_R_MMQ)
#define LOAD_VEC_B 16
@@ -4,6 +4,19 @@
#include "types.glsl"
int32_t mmq_dot(const int32_t a, const int32_t b) {
// With MMQ_SUDOT the A quants must be unsigned
if (MMQ_SUDOT) {
return dotPacked4x8EXT(uint32_t(a), b);
}
return dotPacked4x8EXT(a, b);
}
// With MMQ_SUDOT, flip the sign bit of each byte. This adds a bias of 128 to signed quants.
int32_t mmq_a_store(const int32_t v) {
return MMQ_SUDOT ? v ^ int32_t(0x80808080) : v;
}
// Each iqs value maps to a 32-bit integer
#if defined(DATA_A_Q2_0)
@@ -41,7 +54,7 @@ void block_a_to_registers(const uint reg_ib, const uint buf_ib) {
ACC_TYPE mmq_dot_product(const uint ib_a) {
int32_t q_sum = 0;
[[unroll]] for (uint iqs = 0; iqs < 8; ++iqs) {
q_sum += dotPacked4x8EXT(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
q_sum += mmq_dot(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
}
return ACC_TYPE(float(cache_a[ib_a].dm) * (float(q_sum) * float(cache_b.ds.x) - float(cache_b.ds.y)));
@@ -102,8 +115,8 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
const int32_t qs_b0 = cache_b.qs[iqs];
const int32_t qs_b1 = cache_b.qs[iqs + 4];
q_sum += dotPacked4x8EXT(qs_a.x, qs_b0);
q_sum += dotPacked4x8EXT(qs_a.y, qs_b1);
q_sum += mmq_dot(qs_a.x, qs_b0);
q_sum += mmq_dot(qs_a.y, qs_b1);
}
#ifdef DATA_A_Q4_0
@@ -176,8 +189,8 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
const int32_t qs_b0 = cache_b.qs[iqs];
const int32_t qs_b1 = cache_b.qs[iqs + 4];
q_sum += dotPacked4x8EXT(qs_a0, qs_b0);
q_sum += dotPacked4x8EXT(qs_a1, qs_b1);
q_sum += mmq_dot(qs_a0, qs_b0);
q_sum += mmq_dot(qs_a1, qs_b1);
}
#ifdef DATA_A_Q5_0
@@ -192,8 +205,8 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
// 2-byte loads for Q8_0 blocks (34 bytes)
void block_a_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bool is_in_bounds) {
if (is_in_bounds) {
buf_a[buf_ib].qs[iqs] = pack32(i16vec2(data_a_packed16[ib].qs[iqs * 2],
data_a_packed16[ib].qs[iqs * 2 + 1]));
buf_a[buf_ib].qs[iqs] = mmq_a_store(pack32(i16vec2(data_a_packed16[ib].qs[iqs * 2],
data_a_packed16[ib].qs[iqs * 2 + 1])));
if (iqs == 0) {
buf_a[buf_ib].dm = FLOAT_TYPE(data_a_packed16[ib].d);
@@ -221,7 +234,10 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
const int32_t qs_a = cache_a[ib_a].qs[iqs];
const int32_t qs_b = cache_b.qs[iqs];
q_sum += dotPacked4x8EXT(qs_a, qs_b);
q_sum += mmq_dot(qs_a, qs_b);
}
if (MMQ_SUDOT) {
q_sum -= 128 * (cache_b_qsum[0] + cache_b_qsum[1]);
}
return ACC_TYPE(float(q_sum) * float(cache_a[ib_a].dm) * float(cache_b.ds.x));
@@ -240,8 +256,8 @@ void block_a_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
const u8vec4 i_a0 = unpack8( qs & 0x0F0F0F0F);
const u8vec4 i_a1 = unpack8((qs >> 4) & 0x0F0F0F0F);
buf_a[buf_ib].qs[iqs ] = pack32(i8vec4(kvalues_mxfp4[i_a0.x], kvalues_mxfp4[i_a0.y], kvalues_mxfp4[i_a0.z], kvalues_mxfp4[i_a0.w]));
buf_a[buf_ib].qs[iqs + 4] = pack32(i8vec4(kvalues_mxfp4[i_a1.x], kvalues_mxfp4[i_a1.y], kvalues_mxfp4[i_a1.z], kvalues_mxfp4[i_a1.w]));
buf_a[buf_ib].qs[iqs ] = mmq_a_store(pack32(i8vec4(kvalues_mxfp4[i_a0.x], kvalues_mxfp4[i_a0.y], kvalues_mxfp4[i_a0.z], kvalues_mxfp4[i_a0.w])));
buf_a[buf_ib].qs[iqs + 4] = mmq_a_store(pack32(i8vec4(kvalues_mxfp4[i_a1.x], kvalues_mxfp4[i_a1.y], kvalues_mxfp4[i_a1.z], kvalues_mxfp4[i_a1.w])));
if (iqs == 0) {
buf_a[buf_ib].d = FLOAT_TYPE(e8m0_to_fp32(data_a[ib].e) * 0.5);
@@ -269,7 +285,10 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
[[unroll]] for (uint iqs = 0; iqs < 8; iqs++) {
const int32_t qs_a = cache_a[ib_a].qs[iqs];
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 128 * (cache_b_qsum[0] + cache_b_qsum[1]);
}
return ACC_TYPE(float(cache_a[ib_a].d) * float(cache_b.ds.x) * float(q_sum));
@@ -284,8 +303,8 @@ void block_a_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
const uint32_t vui = data_a_packed32[ib_k].qs[4 * ib32 + iqs];
const i32vec2 qs = iq4nl_to_i8x8(vui);
buf_a[buf_ib].qs[iqs ] = qs.x;
buf_a[buf_ib].qs[iqs + 4] = qs.y;
buf_a[buf_ib].qs[iqs ] = mmq_a_store(qs.x);
buf_a[buf_ib].qs[iqs + 4] = mmq_a_store(qs.y);
if (iqs == 0) {
const uint sl = (data_a_packed32[ib_k].scales_l >> (4 * ib32)) & 0xF;
@@ -313,7 +332,10 @@ void block_a_to_registers(const uint reg_ib, const uint buf_ib) {
ACC_TYPE mmq_dot_product(const uint ib_a) {
int32_t q_sum = 0;
[[unroll]] for (uint iqs = 0; iqs < 8; iqs++) {
q_sum += dotPacked4x8EXT(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
q_sum += mmq_dot(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 128 * (cache_b_qsum[0] + cache_b_qsum[1]);
}
return ACC_TYPE(float(cache_a[ib_a].d) * float(cache_b.ds.x) * float(q_sum));
@@ -372,8 +394,8 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
const int32_t scale_m = int32_t(scale >> 4) * 0x01010101; // Duplicate 8-bit value across 32-bits.
const int32_t qs_a = int32_t((cache_a[ib_a].qs[iqs / 4] >> ((iqs % 4) * 2)) & 0x03030303);
sum_d += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]) * (scale & 0xF);
sum_m += dotPacked4x8EXT(scale_m, cache_b.qs[iqs]);
sum_d += mmq_dot(qs_a, cache_b.qs[iqs]) * (scale & 0xF);
sum_m += mmq_dot(scale_m, cache_b.qs[iqs]);
}
return ACC_TYPE(float(cache_b.ds.x) * (float(cache_a[ib_a].dm.x) * float(sum_d) - float(cache_a[ib_a].dm.y) * float(sum_m)));
@@ -435,18 +457,26 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
int32_t q_sum = 0;
[[unroll]] for (uint iqs = 0; iqs < 4; iqs++) {
// Subtract 4 from the quants to correct the 3rd bit offset
const int32_t qs_a = pack32(unpack8(int32_t((cache_a[ib_a].qs[iqs / 2] >> ((iqs % 2) * 4)) & 0x0F0F0F0F)) - int8_t(4));
// Subtract 4 from the quants to correct the 3rd bit offset. MMQ_SUDOT keeps them unsigned and subtracts 4 times the B sum.
const int32_t qs_u = int32_t((cache_a[ib_a].qs[iqs / 2] >> ((iqs % 2) * 4)) & 0x0F0F0F0F);
const int32_t qs_a = MMQ_SUDOT ? qs_u : pack32(unpack8(qs_u) - int8_t(4));
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 4 * cache_b_qsum[0];
}
result += float(cache_a[ib_a].d_scales[0]) * float(q_sum);
q_sum = 0;
[[unroll]] for (uint iqs = 4; iqs < 8; iqs++) {
const int32_t qs_a = pack32(unpack8(int32_t((cache_a[ib_a].qs[iqs / 2] >> ((iqs % 2) * 4)) & 0x0F0F0F0F)) - int8_t(4));
const int32_t qs_u = int32_t((cache_a[ib_a].qs[iqs / 2] >> ((iqs % 2) * 4)) & 0x0F0F0F0F);
const int32_t qs_a = MMQ_SUDOT ? qs_u : pack32(unpack8(qs_u) - int8_t(4));
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 4 * cache_b_qsum[1];
}
result += float(cache_a[ib_a].d_scales[1]) * float(q_sum);
@@ -518,7 +548,7 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
const int32_t qs_a = cache_a[ib_a].qs[iqs];
#endif
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
return ACC_TYPE(float(cache_b.ds.x) * float(cache_a[ib_a].dm.x) * float(q_sum) - float(cache_a[ib_a].dm.y) * float(cache_b.ds.y));
@@ -539,10 +569,11 @@ void block_a_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
const uint qh_shift = ((iqs_k % 32) / 8) * 2;
const i8vec2 vals00 = (unpack8(int32_t((data_a_packed16[ib_k].ql[ql_idx * 2 ] >> ql_shift) & uint16_t(0x0F0F))).xy |
unpack8(int32_t(((data_a_packed16[ib_k].qh[qh_idx * 2 ] >> qh_shift) & uint16_t(0x0303)) << 4)).xy) - int8_t(32);
unpack8(int32_t(((data_a_packed16[ib_k].qh[qh_idx * 2 ] >> qh_shift) & uint16_t(0x0303)) << 4)).xy);
const i8vec2 vals01 = (unpack8(int32_t((data_a_packed16[ib_k].ql[ql_idx * 2 + 1] >> ql_shift) & uint16_t(0x0F0F))).xy |
unpack8(int32_t(((data_a_packed16[ib_k].qh[qh_idx * 2 + 1] >> qh_shift) & uint16_t(0x0303)) << 4)).xy) - int8_t(32);
buf_a[buf_ib].qs[iqs] = pack32(i8vec4(vals00.x, vals00.y, vals01.x, vals01.y));
unpack8(int32_t(((data_a_packed16[ib_k].qh[qh_idx * 2 + 1] >> qh_shift) & uint16_t(0x0303)) << 4)).xy);
// MMQ_SUDOT keeps the quants unsigned and subtracts 32 times the B sum
buf_a[buf_ib].qs[iqs] = MMQ_SUDOT ? pack32(i8vec4(vals00, vals01)) : pack32(i8vec4(vals00 - int8_t(32), vals01 - int8_t(32)));
if (iqs == 0) {
const uint is = iqs_k / 4;
@@ -574,7 +605,10 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
[[unroll]] for (uint iqs = 0; iqs < 4; iqs++) {
const int32_t qs_a = cache_a[ib_a].qs[iqs];
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 32 * cache_b_qsum[0];
}
result += float(cache_a[ib_a].d_scales[0]) * float(q_sum);
q_sum = 0;
@@ -582,7 +616,10 @@ ACC_TYPE mmq_dot_product(const uint ib_a) {
[[unroll]] for (uint iqs = 4; iqs < 8; iqs++) {
const int32_t qs_a = cache_a[ib_a].qs[iqs];
q_sum += dotPacked4x8EXT(qs_a, cache_b.qs[iqs]);
q_sum += mmq_dot(qs_a, cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 32 * cache_b_qsum[1];
}
result += float(cache_a[ib_a].d_scales[1]) * float(q_sum);
@@ -612,8 +649,8 @@ void block_a_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
const ivec4 m0 = -(ivec4(signs, signs >> 1, signs >> 2, signs >> 3) & 1);
const ivec4 m1 = -(ivec4(signs >> 4, signs >> 5, signs >> 6, signs >> 7) & 1);
buf_a[buf_ib].qs[2 * iqs ] = pack32(i8vec4((vals0 ^ m0) - m0));
buf_a[buf_ib].qs[2 * iqs + 1] = pack32(i8vec4((vals1 ^ m1) - m1));
buf_a[buf_ib].qs[2 * iqs ] = mmq_a_store(pack32(i8vec4((vals0 ^ m0) - m0)));
buf_a[buf_ib].qs[2 * iqs + 1] = mmq_a_store(pack32(i8vec4((vals1 ^ m1) - m1)));
if (iqs == 0) {
const uint scale = (uint(data_a_packed16[ib_k].scales[ib32 / 4]) >> ((ib32 & 3) * 4)) & 0xF;
@@ -641,7 +678,10 @@ void block_a_to_registers(const uint reg_ib, const uint buf_ib) {
ACC_TYPE mmq_dot_product(const uint ib_a) {
int32_t q_sum = 0;
[[unroll]] for (uint iqs = 0; iqs < 8; iqs++) {
q_sum += dotPacked4x8EXT(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
q_sum += mmq_dot(cache_a[ib_a].qs[iqs], cache_b.qs[iqs]);
}
if (MMQ_SUDOT) {
q_sum -= 128 * (cache_b_qsum[0] + cache_b_qsum[1]);
}
return ACC_TYPE(float(cache_a[ib_a].d) * float(cache_b.ds.x) * float(q_sum));
@@ -662,6 +702,12 @@ void block_b_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
buf_b[buf_ib].qs[iqs * 4 + 1] = values.y;
buf_b[buf_ib].qs[iqs * 4 + 2] = values.z;
buf_b[buf_ib].qs[iqs * 4 + 3] = values.w;
#ifdef MMQ_B_SUMS
if (MMQ_SUDOT) {
buf_b_qsum[buf_ib * 2 + iqs] = dotPacked4x8EXT(values.x, 0x01010101u) + dotPacked4x8EXT(values.y, 0x01010101u) +
dotPacked4x8EXT(values.z, 0x01010101u) + dotPacked4x8EXT(values.w, 0x01010101u);
}
#endif
} else {
if (iqs == 0) {
buf_b[buf_ib].ds = FLOAT_TYPEV2(0.0f);
@@ -671,6 +717,11 @@ void block_b_to_shmem(const uint buf_ib, const uint ib, const uint iqs, const bo
buf_b[buf_ib].qs[iqs * 4 + 1] = 0;
buf_b[buf_ib].qs[iqs * 4 + 2] = 0;
buf_b[buf_ib].qs[iqs * 4 + 3] = 0;
#ifdef MMQ_B_SUMS
if (MMQ_SUDOT) {
buf_b_qsum[buf_ib * 2 + iqs] = 0;
}
#endif
}
}
@@ -679,4 +730,10 @@ void block_b_to_registers(const uint ib) {
[[unroll]] for (uint iqs = 0; iqs < BK / 4; iqs++) {
cache_b.qs[iqs] = buf_b[ib].qs[iqs];
}
#ifdef MMQ_B_SUMS
if (MMQ_SUDOT) {
cache_b_qsum[0] = buf_b_qsum[ib * 2];
cache_b_qsum[1] = buf_b_qsum[ib * 2 + 1];
}
#endif
}
@@ -0,0 +1,37 @@
#version 450
#extension GL_EXT_shader_realtime_clock : require
layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
// never written, it only anchors the spin below so the compiler cannot drop it
layout (binding = 0) writeonly buffer D {uint data_d[];};
layout (push_constant) uniform parameter {
uint ticks;
} p;
// bounds the spin in case the clock does not advance, turning a hang into a too-short sleep
const uint MAX_ITER = 1u << 27;
void main() {
const uvec2 t0 = clockRealtime2x32EXT();
uvec2 t = t0;
for (uint iter = 0; iter < MAX_ITER; ++iter) {
t = clockRealtime2x32EXT();
const uint lo = t.x - t0.x;
const uint hi = t.y - t0.y - (t.x < t0.x ? 1u : 0u);
if (hi != 0u || lo >= p.ticks) {
break;
}
}
// the store never runs, but the compiler cannot prove it and has to keep the spin
if (t.x == 0xFFFFFFFFu && t.y == 0xFFFFFFFFu) {
data_d[0] = t.x;
}
}
+1 -1
View File
@@ -36,6 +36,6 @@ void main() {
if (pass) {
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(data_a[get_aoffset() + src0_idx(idx)]);
} else {
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(0);
data_d[get_doffset() + dst_idx(idx)] = D_TYPE(0.0);
}
}
@@ -948,6 +948,7 @@ void process_shaders() {
string_to_spv("acc_f32", "acc.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
string_to_spv("split_k_reduce", "mul_mat_split_k_reduce.comp", {});
string_to_spv("stream_k_fixup", "mul_mat_stream_k_fixup.comp", {});
string_to_spv("fa_split_k_reduce", "flash_attn_split_k_reduce.comp", {});
string_to_spv("fa_mask_opt", "flash_attn_mask_opt.comp", {});
@@ -1027,6 +1028,7 @@ void process_shaders() {
string_to_spv("arange_f32", "arange.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
string_to_spv("fill_f32", "fill.comp", {{"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
string_to_spv("fill_f16", "fill.comp", {{"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}});
string_to_spv("sleep", "sleep.comp", {});
string_to_spv("silu_back_f32", "silu_back.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}});
@@ -1272,6 +1272,7 @@ class ggml_webgpu_shader_lib {
std::unordered_map<int, webgpu_pipeline> arange_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> count_equal_pipelines; // key is src_overlap
std::unordered_map<int, webgpu_pipeline> timestep_embedding_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> pool2d_pipelines; // key is ggml_op_pool
std::unordered_map<ggml_webgpu_row_norm_pipeline_key, webgpu_pipeline, ggml_webgpu_row_norm_pipeline_key_hash>
row_norm_pipelines; // op/inplace
@@ -2067,6 +2068,34 @@ class ggml_webgpu_shader_lib {
return arange_pipelines[1];
}
webgpu_pipeline get_pool2d_pipeline(const ggml_webgpu_shader_lib_context & context) {
const int key = ggml_get_op_params_i32(context.dst, 0);
auto it = pool2d_pipelines.find(key);
if (it != pool2d_pipelines.end()) {
return it->second;
}
std::vector<std::string> defines;
std::string variant = "pool2d";
if (key == GGML_OP_POOL_MAX) {
defines.push_back("POOL_MAX");
variant += "_max";
} else {
variant += "_avg";
}
defines.push_back(std::string("WG_SIZE=") + std::to_string(context.max_wg_size));
auto processed = preprocessor.preprocess(wgsl_pool2d, defines);
auto decisions = std::make_shared<ggml_webgpu_generic_shader_decisions>();
decisions->wg_size = context.max_wg_size;
webgpu_pipeline pipeline = ggml_webgpu_create_pipeline(device, processed, variant);
pipeline.context = decisions;
pool2d_pipelines[key] = pipeline;
return pool2d_pipelines[key];
}
webgpu_pipeline get_pad_pipeline(const ggml_webgpu_shader_lib_context & context) {
ggml_webgpu_pad_pipeline_key key = {};
key.reflect = context.dst->op == GGML_OP_PAD_REFLECT_1D;
+45
View File
@@ -961,6 +961,46 @@ static webgpu_encoded_op ggml_webgpu_pad(webgpu_context & ctx, ggml_tensor * src
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_pool2d(webgpu_context & ctx, ggml_tensor * src, ggml_tensor * dst) {
ggml_webgpu_shader_lib_context shader_lib_ctx = {};
shader_lib_ctx.src0 = src;
shader_lib_ctx.dst = dst;
shader_lib_ctx.max_wg_size = ctx->global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
webgpu_pipeline pipeline = ctx->shader_lib->get_pool2d_pipeline(shader_lib_ctx);
auto * decisions = static_cast<ggml_webgpu_generic_shader_decisions *>(pipeline.context.get());
const uint32_t ne = (uint32_t) ggml_nelements(dst);
std::vector<uint32_t> params = {
ne,
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, src) / ggml_type_size(src->type)),
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, dst) / ggml_type_size(dst->type)),
(uint32_t) src->ne[0],
(uint32_t) src->ne[1],
(uint32_t) dst->ne[0],
(uint32_t) dst->ne[1],
(uint32_t) ggml_get_op_params_i32(dst, 1), // k0
(uint32_t) ggml_get_op_params_i32(dst, 2), // k1
(uint32_t) ggml_get_op_params_i32(dst, 3), // s0
(uint32_t) ggml_get_op_params_i32(dst, 4), // s1
(uint32_t) ggml_get_op_params_i32(dst, 5), // p0
(uint32_t) ggml_get_op_params_i32(dst, 6), // p1
};
std::vector<wgpu::BindGroupEntry> entries = {
ggml_webgpu_make_tensor_bind_group_entry(ctx, 0, src),
ggml_webgpu_make_tensor_bind_group_entry(ctx, 1, dst),
};
uint32_t wg_x;
uint32_t wg_y;
uint32_t total_wg = CEIL_DIV(ne, decisions->wg_size);
compute_2d_workgroups(total_wg, ctx->global_ctx->capabilities.limits.maxComputeWorkgroupsPerDimension, wg_x, wg_y);
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_solve_tri(webgpu_context & ctx,
ggml_tensor * src0,
ggml_tensor * src1,
@@ -3670,6 +3710,8 @@ static std::optional<webgpu_encoded_op> ggml_webgpu_encode(webgpu_context ctx,
}
case GGML_OP_ARANGE:
return ggml_webgpu_arange(ctx, node);
case GGML_OP_POOL_2D:
return ggml_webgpu_pool2d(ctx, src0, node);
case GGML_OP_PAD_REFLECT_1D:
case GGML_OP_PAD:
return ggml_webgpu_pad(ctx, src0, node);
@@ -5219,6 +5261,9 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
case GGML_OP_ARANGE:
supports_op = op->type == GGML_TYPE_F32 && ggml_is_contiguous(op);
break;
case GGML_OP_POOL_2D:
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32 && ggml_is_contiguous(src0);
break;
case GGML_OP_PAD_REFLECT_1D:
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32;
break;
@@ -0,0 +1,74 @@
@group(0) @binding(0)
var<storage, read_write> src: array<f32>;
@group(0) @binding(1)
var<storage, read_write> dst: array<f32>;
struct Params {
ne: u32, // total number of elements
offset_src: u32, // in elements
offset_dst: u32, // in elements
// Logical shapes
src_ne0: u32,
src_ne1: u32,
dst_ne0: u32,
dst_ne1: u32,
k0: u32,
k1: u32,
s0: u32,
s1: u32,
p0: u32,
p1: u32
}
@group(0) @binding(2)
var<uniform> params: Params;
@compute @workgroup_size(WG_SIZE)
fn main(@builtin(num_workgroups) num_wg: vec3<u32>,
@builtin(global_invocation_id) gid: vec3<u32>) {
let gid_i = gid.x + (num_wg.x * u32(WG_SIZE)) * gid.y;
if (gid_i >= params.ne) {
return;
}
let out_x = gid_i % params.dst_ne0;
let out_y = (gid_i / params.dst_ne0) % params.dst_ne1;
let out_z = (gid_i / (params.dst_ne0 * params.dst_ne1));
let x0 = i32(out_x * params.s0) - i32(params.p0);
let y0 = i32(out_y * params.s1) - i32(params.p1);
let in_x_start = max(0, x0);
let in_x_end = min(i32(params.src_ne0), x0 + i32(params.k0));
let in_y_start = max(0, y0);
let in_y_end = min(i32(params.src_ne1), y0 + i32(params.k1));
let base = params.offset_src + out_z * params.src_ne0 * params.src_ne1;
#ifdef POOL_MAX
var res: f32 = bitcast<f32>(0xff7fffffu);
#else
var res: f32 = 0.0;
#endif
for(var y = in_y_start; y < in_y_end; y++){
let row = base + u32(y) * params.src_ne0;
for(var x = in_x_start; x < in_x_end; x++){
#ifdef POOL_MAX
res = max(res, src[row + u32(x)]);
#else
res += src[row + u32(x)];
#endif
}
}
#ifndef POOL_MAX
res = res / f32(params.k0 * params.k1);
#endif
dst[params.offset_dst + gid_i] = res;
}
+27 -2
View File
@@ -1112,9 +1112,11 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
"OPT_STEP_SGD",
"GLU",
"SLEEP",
};
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT != 99");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"none",
@@ -1225,9 +1227,11 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"sgd(x)",
"glu(x)",
"sleep(x)",
};
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT != 99");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
@@ -6176,6 +6180,27 @@ struct ggml_tensor * ggml_custom_inplace(
return result;
}
// ggml_sleep
// the result aliases a, the op only delays and never touches the data
struct ggml_tensor * ggml_sleep(
struct ggml_context * ctx,
struct ggml_tensor * a,
int32_t us) {
GGML_ASSERT(us >= 0);
struct ggml_tensor * result = ggml_view_tensor(ctx, a);
ggml_format_name(result, "%s (sleep)", a->name);
ggml_set_op_params_i32(result, 0, us);
result->op = GGML_OP_SLEEP;
result->src[0] = a;
return result;
}
// ggml_cross_entropy_loss
struct ggml_tensor * ggml_cross_entropy_loss(
+8 -5
View File
@@ -41,7 +41,7 @@ function(mime_from_ext name out_var)
set(m "text/html; charset=utf-8")
elseif(ext_str STREQUAL "css")
set(m "text/css")
elseif(ext_str STREQUAL "js")
elseif(ext_str STREQUAL "js" OR ext_str STREQUAL "mjs")
set(m "application/javascript")
elseif(ext_str STREQUAL "json")
set(m "application/json")
@@ -98,9 +98,12 @@ function(ui_validate_assets files in_dir)
set(found_build_json TRUE)
elseif(base STREQUAL "version.json")
set(found_version_json TRUE)
elseif(base MATCHES "^bundle.*\\.js$")
# the app's entry: bundle[hash].js/.css from a single-bundle build (builds before the
# split one, prebuilt archives included), or entry/start[hash].js and the CSS in
# _app/immutable/assets from a split build
elseif(base MATCHES "^bundle.*\\.js$" OR f MATCHES "(^|/)_app/immutable/entry/start\\.[^/]*\\.js$")
set(found_bundle_js TRUE)
elseif(base MATCHES "^bundle.*\\.css$")
elseif(base MATCHES "^bundle.*\\.css$" OR f MATCHES "(^|/)_app/immutable/assets/[^/]*\\.css$")
set(found_bundle_css TRUE)
endif()
endforeach()
@@ -125,10 +128,10 @@ function(ui_validate_assets files in_dir)
list(APPEND missing "version.json")
endif()
if(NOT found_bundle_js)
list(APPEND missing "bundle[hash].js")
list(APPEND missing "_app/immutable/entry/start[hash].js (or bundle[hash].js)")
endif()
if(NOT found_bundle_css)
list(APPEND missing "bundle[hash].css")
list(APPEND missing "_app/immutable/assets/[name][hash].css (or bundle[hash].css)")
endif()
if(missing)
-3
View File
@@ -3095,7 +3095,6 @@ llm_graph_params llama_context::graph_params(
llm_graph_type gtype) const {
return {
/*.model =*/ &model,
/*.hparams =*/ model.hparams,
/*.cparams =*/ cparams,
/*.ubatch =*/ ubatch,
/*.gtype =*/ gtype,
@@ -3106,8 +3105,6 @@ llm_graph_params llama_context::graph_params(
/*.mctx =*/ mctx,
/*.cross =*/ &cross,
/*.moe_cache =*/ moe_cache.get(),
/*.prec_policy =*/ &model.prec_policy,
/*.hdmd =*/ model.hdmd.rot.empty() ? nullptr : &model.hdmd,
/*.layer_inp_dev =*/ layer_inp_dev,
/*.layer_inp_src =*/ supports_embd_row() ? cparams.ctx_other->get_layer_inp_dev() : nullptr,
/*.h_nextn_dev =*/ cparams.embeddings_nextn_dev ? h_nextn_dev : nullptr,
+213 -34
View File
@@ -1400,6 +1400,91 @@ bool llm_graph_input_mem_hybrid_iswa::can_reuse(const llm_graph_params & params)
return res;
}
void llm_graph_input_mem_hybrid_iswa_rel::set_input(const llama_ubatch * ubatch) {
const auto * attn_ctx = mctx->get_attn();
// base tensors may not be allocated if there are no non-SWA attention layers
if (inp_attn->self_k_idxs && inp_attn->self_k_idxs->buffer) {
attn_ctx->get_base()->set_input_k_idxs(inp_attn->self_k_idxs, ubatch);
attn_ctx->get_base()->set_input_v_idxs(inp_attn->self_v_idxs, ubatch);
}
if (inp_attn->self_attn_mask.raw && inp_attn->self_attn_mask.raw->buffer) {
attn_ctx->get_base()->set_input_attn_mask(inp_attn->self_attn_mask, ubatch, cparams.causal_attn);
}
if (self_rel_pos->buffer) {
attn_ctx->get_base()->set_input_relative_pos(self_rel_pos, ubatch, extent);
}
// swa tensors may not be allocated if there are no SWA attention layers
if (inp_attn->self_k_idxs_swa && inp_attn->self_k_idxs_swa->buffer) {
attn_ctx->get_swa()->set_input_k_idxs(inp_attn->self_k_idxs_swa, ubatch);
attn_ctx->get_swa()->set_input_v_idxs(inp_attn->self_v_idxs_swa, ubatch);
}
if (inp_attn->self_attn_mask_swa.raw && inp_attn->self_attn_mask_swa.raw->buffer) {
attn_ctx->get_swa()->set_input_attn_mask(inp_attn->self_attn_mask_swa, ubatch, cparams.causal_attn);
}
if (self_rel_pos_swa->buffer) {
attn_ctx->get_swa()->set_input_relative_pos(self_rel_pos_swa, ubatch, extent_swa);
}
const int64_t n_rs = mctx->get_recr()->get_n_rs();
if (inp_rs->s_copy) {
GGML_ASSERT(ggml_backend_buffer_is_host(inp_rs->s_copy->buffer));
int32_t * data = (int32_t *) inp_rs->s_copy->data;
// assuming copy destinations ALWAYS happen ONLY on the cells between head and head+n
for (uint32_t i = 0; i < n_rs; ++i) {
data[i] = mctx->get_recr()->s_copy(i);
}
}
}
bool llm_graph_input_mem_hybrid_iswa_rel::can_reuse(const llm_graph_params & params) {
const auto * mctx = static_cast<const llama_memory_hybrid_iswa_context *>(params.mctx);
this->mctx = mctx;
bool res = true;
const auto * attn_ctx = mctx->get_attn();
// the masks are always dense, see build_inp_mem_hybrid_iswa_rel()
// base tensors may not be allocated if there are no non-SWA attention layers
if (inp_attn->self_k_idxs && inp_attn->self_k_idxs->buffer) {
res &= inp_attn->self_k_idxs->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask.raw, attn_ctx->get_base(), params.ubatch, params.cparams);
res &= self_rel_pos->ne[0] == attn_ctx->get_base()->get_n_kv();
res &= self_rel_pos->ne[1] == params.ubatch.n_tokens;
// swa tensors may not be allocated if there are no SWA attention layers
if (inp_attn->self_k_idxs_swa && inp_attn->self_k_idxs_swa->buffer) {
res &= inp_attn->self_k_idxs_swa->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask_swa.raw, attn_ctx->get_swa(), params.ubatch, params.cparams);
res &= self_rel_pos_swa->ne[0] == attn_ctx->get_swa()->get_n_kv();
res &= self_rel_pos_swa->ne[1] == params.ubatch.n_tokens;
res &= inp_rs->s_copy->ne[0] == mctx->get_recr()->get_n_rs();
res &= inp_rs->s_copy_main->ne[0] == params.ubatch.n_seqs;
res &= inp_rs->head == mctx->get_recr()->get_head();
res &= inp_rs->rs_z == mctx->get_recr()->get_rs_z();
return res;
}
void llm_graph_input_sampling::set_input(const llama_ubatch * ubatch) {
// set the inputs only for the active samplers in the current ubatch
std::unordered_set<llama_seq_id> active_samplers;
@@ -1594,9 +1679,10 @@ void llm_graph_result::set_params(const llm_graph_params & params) {
llm_graph_context::llm_graph_context(const llm_graph_params & params) :
model (*params.model),
hparams (params.hparams),
hparams (model.hparams),
cparams (params.cparams),
ubatch (params.ubatch),
prec_policy (model.prec_policy),
n_embd (hparams.n_embd),
n_layer (hparams.n_layer()),
n_layer_nextn (hparams.n_layer_nextn),
@@ -1630,8 +1716,7 @@ llm_graph_context::llm_graph_context(const llm_graph_params & params) :
mctx (params.mctx),
cross (params.cross),
moe_cache (params.moe_cache),
prec_policy (params.prec_policy),
hdmd (params.hdmd),
hdmd (model.hdmd.rot.empty() ? nullptr : &model.hdmd),
layer_inp_dev (params.layer_inp_dev),
layer_inp_src (params.layer_inp_src),
h_nextn_dev (params.h_nextn_dev),
@@ -1728,9 +1813,7 @@ ggml_tensor * llm_graph_context::build_lora_mm(
ggml_tensor * cur_mm = build_hadamard_input(w, cur);
ggml_tensor * res = ggml_mul_mat(ctx0, w, cur_mm);
if (prec_policy) {
prec_policy->apply(res);
}
prec_policy.apply(res);
if (w->type == GGML_TYPE_NVFP4) {
ggml_prec_set_acc(res, GGML_PREC_BF16);
@@ -1812,9 +1895,7 @@ ggml_tensor * llm_graph_context::build_lora_mm_id(
ggml_mul_mat_id(ctx0, w, cur_mm, ids) :
ggml_mul_mat_id(ctx0, cached ? moe_cache->get_experts(w) : w, cur_mm, slots);
if (prec_policy) {
prec_policy->apply(res);
}
prec_policy.apply(res);
if (w->type == GGML_TYPE_NVFP4) {
ggml_prec_set_acc(res, GGML_PREC_BF16);
@@ -3117,7 +3198,7 @@ ggml_tensor * llm_graph_context::build_attn_mha(
ggml_tensor * cur;
const bool use_flash_attn = cparams.flash_attn && kq_b == nullptr && (!sinks || ggml_is_vector(sinks));
const bool use_flash_attn = cparams.flash_attn && kq_b == nullptr;
GGML_ASSERT(use_flash_attn || kv_rows == nullptr);
if (use_flash_attn) {
GGML_ASSERT(kq_b == nullptr && "Flash attention does not support KQ bias yet");
@@ -3200,16 +3281,8 @@ ggml_tensor * llm_graph_context::build_attn_mha(
cb(kq, "kq_plus_kq_b", il);
}
if (sinks && !ggml_is_vector(sinks)) {
// Per-query sinks are extra logits with zero values.
const int64_t n_kv = kq->ne[0];
kq = ggml_add(ctx0, ggml_scale(ctx0, kq, kq_scale), ggml_cast(ctx0, kq_mask, GGML_TYPE_F32));
kq = ggml_soft_max(ctx0, ggml_concat(ctx0, kq, sinks, 0));
kq = ggml_cont(ctx0, ggml_view_4d(ctx0, kq, n_kv, kq->ne[1], kq->ne[2], kq->ne[3], kq->nb[1], kq->nb[2], kq->nb[3], 0));
} else {
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);
ggml_soft_max_add_sinks(kq, sinks);
}
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);
ggml_soft_max_add_sinks(kq, sinks);
cb(kq, "kq_soft_max", il);
if (!v_trans) {
@@ -3691,7 +3764,9 @@ ggml_tensor * llm_graph_context::build_attn(
return cur;
}
ggml_tensor * llm_graph_context::build_attn(
// the iswa build_attn with the KQ mask and kv_rows given by the caller, which has to keep them in sync
static ggml_tensor * build_attn_iswa_impl(
const llm_graph_context & gctx,
llm_graph_input_attn_kv_iswa * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
@@ -3700,11 +3775,18 @@ ggml_tensor * llm_graph_context::build_attn(
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * kq_mask,
ggml_tensor * kv_rows,
ggml_tensor * sinks,
ggml_tensor * v_mla,
float kq_scale,
int il,
ggml_tensor * kq_mask) const {
int il) {
ggml_context * ctx0 = gctx.ctx0;
ggml_cgraph * gf = gctx.gf;
const auto & hparams = gctx.hparams;
const auto & cparams = gctx.cparams;
const bool is_swa = hparams.is_swa(il);
auto * k_rot = is_swa ? inp->self_k_rot_swa[il] : inp->self_k_rot[il];
@@ -3742,7 +3824,7 @@ ggml_tensor * llm_graph_context::build_attn(
// whole seq fits into batch in training mode
const bool use_kv_cur = cparams.training && k_cur && v_cur;
if (use_kv_cur) {
GGML_ASSERT(mctx_cur->get_n_kv() == n_tokens);
GGML_ASSERT(mctx_cur->get_n_kv() == gctx.n_tokens);
}
const bool store_kv = !use_kv_cur || hparams.n_layer_kv_from_start >= 0;
@@ -3760,25 +3842,20 @@ ggml_tensor * llm_graph_context::build_attn(
ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
}
if (!kq_mask) {
kq_mask = is_swa ? inp->self_attn_mask_swa.raw : inp->self_attn_mask.raw;
}
const auto & kv_rows = is_swa ? inp->self_attn_mask_swa.rows : inp->self_attn_mask.rows;
ggml_tensor * q = q_cur;
ggml_tensor * k = use_kv_cur ? k_cur : mctx_cur->get_k(ctx0, il);
ggml_tensor * v = use_kv_cur ? v_cur : mctx_cur->get_v(ctx0, il);
ggml_tensor * cur = build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il);
ggml_tensor * cur = gctx.build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il);
cur = build_attn_scale_kv_output(ctx0, mctx_cur, cur, il);
cb(cur, "kqv_out", il);
gctx.cb(cur, "kqv_out", il);
if (v_rot) {
cur = llama_mul_mat_hadamard(ctx0, cur, v_rot);
}
if (wo) {
cur = build_lora_mm(wo, cur, wo_s);
cur = gctx.build_lora_mm(wo, cur, wo_s);
}
if (wo_b) {
@@ -3792,6 +3869,27 @@ ggml_tensor * llm_graph_context::build_attn(
return cur;
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_attn_kv_iswa * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur,
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * sinks,
ggml_tensor * v_mla,
float kq_scale,
int il) const {
const bool is_swa = hparams.is_swa(il);
const auto & kq_mask = is_swa ? inp->self_attn_mask_swa.raw : inp->self_attn_mask.raw;
const auto & kv_rows = is_swa ? inp->self_attn_mask_swa.rows : inp->self_attn_mask.rows;
return build_attn_iswa_impl(*this, inp, wo, wo_b, wo_s, q_cur, k_cur, v_cur, kq_b, kq_mask, kv_rows, sinks, v_mla, kq_scale, il);
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_attn_k_iswa * inp,
ggml_tensor * wo,
@@ -4310,6 +4408,43 @@ llm_graph_input_mem_hybrid_iswa * llm_graph_context::build_inp_mem_hybrid_iswa()
auto inp_attn = std::make_unique<llm_graph_input_attn_kv_iswa>(hparams, cparams, attn_ctx);
{
inp_attn->self_k_idxs = attn_ctx->get_base()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs = attn_ctx->get_base()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask = build_attn_inp_attn_mask(ctx0, attn_ctx->get_base(), ubatch, cparams);
}
{
inp_attn->self_k_idxs_swa = attn_ctx->get_swa()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs_swa = attn_ctx->get_swa()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask_swa = build_attn_inp_attn_mask(ctx0, attn_ctx->get_swa(), ubatch, cparams);
}
inp_attn->self_k_rot = attn_ctx->get_base()->get_k_rot();
inp_attn->self_v_rot = attn_ctx->get_base()->get_v_rot();
inp_attn->self_k_rot_swa = attn_ctx->get_swa()->get_k_rot();
inp_attn->self_v_rot_swa = attn_ctx->get_swa()->get_v_rot();
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur);
return (llm_graph_input_mem_hybrid_iswa *) res->add_input(std::move(inp));
}
llm_graph_input_mem_hybrid_iswa_rel * llm_graph_context::build_inp_mem_hybrid_iswa_rel(
uint32_t extent,
uint32_t extent_swa) const {
const auto * mctx_cur = static_cast<const llama_memory_hybrid_iswa_context *>(mctx);
auto inp_rs = build_rs_inp_impl(ctx0, ubatch, mctx_cur->get_recr());
const auto * attn_ctx = mctx_cur->get_attn();
auto inp_attn = std::make_unique<llm_graph_input_attn_kv_iswa>(hparams, cparams, attn_ctx);
// dense masks: the bias is gathered by the distance to each cell of the K/V view
{
inp_attn->self_k_idxs = attn_ctx->get_base()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs = attn_ctx->get_base()->build_input_v_idxs(ctx0, ubatch);
@@ -4330,9 +4465,53 @@ llm_graph_input_mem_hybrid_iswa * llm_graph_context::build_inp_mem_hybrid_iswa()
inp_attn->self_k_rot_swa = attn_ctx->get_swa()->get_k_rot();
inp_attn->self_v_rot_swa = attn_ctx->get_swa()->get_v_rot();
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur);
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa_rel>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur, extent, extent_swa);
return (llm_graph_input_mem_hybrid_iswa *) res->add_input(std::move(inp));
inp->self_rel_pos = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_base()->get_n_kv(), ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos);
ggml_set_name(inp->self_rel_pos, "attn_inp_rel_pos");
inp->self_rel_pos_swa = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_swa()->get_n_kv(), ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos_swa);
ggml_set_name(inp->self_rel_pos_swa, "attn_inp_rel_pos_swa");
return (llm_graph_input_mem_hybrid_iswa_rel *) res->add_input(std::move(inp));
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_mem_hybrid_iswa_rel * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur,
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * sinks,
float kq_scale,
int il) const {
const bool is_swa = hparams.is_swa(il);
auto * inp_attn = inp->get_attn();
const auto & attn_mask = is_swa ? inp_attn->self_attn_mask_swa : inp_attn->self_attn_mask;
// TODO: support kv_rows - fill the distances over the kv_rows cell lists and pass the rows to build_attn_mha,
// needs a kv_rows flash attention with a per-head mask (CUDA has none)
GGML_ASSERT(attn_mask.rows == nullptr);
// a mask per head, in the dense layout of the input mask, which flash attention takes in F16
ggml_tensor * kq_mask = attn_mask.raw;
if (kq_mask->type != GGML_TYPE_F32) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F32);
}
kq_mask = ggml_add(ctx0, kq_b, kq_mask);
if (cparams.flash_attn) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F16);
}
cb(kq_mask, "kq_mask_rel_pos", il);
return build_attn_iswa_impl(*this, inp_attn, wo, wo_b, wo_s, q_cur, k_cur, v_cur, nullptr, kq_mask, nullptr, sinks, nullptr, kq_scale, il);
}
void llm_graph_context::build_dense_out(
+64 -9
View File
@@ -774,6 +774,48 @@ public:
const llama_memory_hybrid_iswa_context * mctx;
};
// hybrid iswa memory with a per-head attention bias that depends on the distance from the query to the cell (Inkling)
// the KQ masks are always dense (rows == nullptr) and are filled from the same cells as the distances, so that the bias
// gathered by distance lines up with the mask
class llm_graph_input_mem_hybrid_iswa_rel : public llm_graph_input_i {
public:
llm_graph_input_mem_hybrid_iswa_rel(
const llama_cparams & cparams,
std::unique_ptr<llm_graph_input_attn_kv_iswa> inp_attn,
std::unique_ptr<llm_graph_input_rs> inp_rs,
const llama_memory_hybrid_iswa_context * mctx,
uint32_t extent,
uint32_t extent_swa) :
inp_attn(std::move(inp_attn)),
inp_rs(std::move(inp_rs)),
extent(extent),
extent_swa(extent_swa),
cparams(cparams),
mctx(mctx) { }
virtual ~llm_graph_input_mem_hybrid_iswa_rel() = default;
void set_input(const llama_ubatch * ubatch) override;
bool can_reuse(const llm_graph_params & params) override;
std::unique_ptr<llm_graph_input_attn_kv_iswa> inp_attn;
std::unique_ptr<llm_graph_input_rs> inp_rs;
llm_graph_input_attn_kv_iswa * get_attn() const { return inp_attn.get(); }
llm_graph_input_rs * get_recr() const { return inp_rs.get(); }
// distance from the token to the cell, extent for the empty, future and more distant cells
ggml_tensor * self_rel_pos = nullptr; // I32 [n_kv, n_batch]
ggml_tensor * self_rel_pos_swa = nullptr; // I32 [n_kv_swa, n_batch]
const uint32_t extent;
const uint32_t extent_swa;
const llama_cparams cparams;
const llama_memory_hybrid_iswa_context * mctx;
};
class llm_graph_input_sampling : public llm_graph_input_i {
public:
llm_graph_input_sampling(std::map<llama_seq_id, llama_sampler *> samplers) :
@@ -804,7 +846,6 @@ class llm_graph_result;
struct llm_graph_params {
const llama_model * model = nullptr;
llama_hparams hparams;
llama_cparams cparams;
llama_ubatch ubatch; // note: intentionally make a copy
@@ -820,10 +861,6 @@ struct llm_graph_params {
const llama_cross * cross;
const llama_moe_cache * moe_cache;
const llama_prec_policy * prec_policy = nullptr;
const llama_hadamard * hdmd = nullptr;
ggml_tensor * layer_inp_dev = nullptr; // own layer input device buffer (write)
ggml_tensor * layer_inp_src = nullptr; // ctx_other's layer input device buffer (read)
ggml_tensor * h_nextn_dev = nullptr; // own h_nextn device buffer (write)
@@ -1056,6 +1093,8 @@ struct llm_graph_context {
const llama_cparams & cparams;
const llama_ubatch & ubatch;
const llama_prec_policy & prec_policy;
const int64_t n_embd;
const int64_t n_layer;
const int64_t n_layer_nextn;
@@ -1096,8 +1135,6 @@ struct llm_graph_context {
const llama_cross * cross;
const llama_moe_cache * moe_cache;
const llama_prec_policy * prec_policy;
const llama_hadamard * hdmd;
ggml_tensor * layer_inp_dev;
@@ -1406,8 +1443,7 @@ struct llm_graph_context {
ggml_tensor * sinks, // [n_head_q]
ggml_tensor * v_mla, // [n_embd_head_v_mla, n_embd_head_v, n_head_v]
float kq_scale,
int il,
ggml_tensor * kq_mask = nullptr) const;
int il) const;
llm_graph_input_attn_k_iswa * build_attn_inp_k_iswa() const;
@@ -1500,6 +1536,25 @@ struct llm_graph_context {
llm_graph_input_mem_hybrid_iswa * build_inp_mem_hybrid_iswa() const;
// extent, extent_swa: the distance that the bias tables of the full and the SWA layers stop at
llm_graph_input_mem_hybrid_iswa_rel * build_inp_mem_hybrid_iswa_rel(
uint32_t extent,
uint32_t extent_swa) const;
// kq_b is added to the KQ mask, after the KQ scale
ggml_tensor * build_attn(
llm_graph_input_mem_hybrid_iswa_rel * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur, // [n_embd_head_q, n_head_q, n_tokens]
ggml_tensor * k_cur, // [n_embd_head_k, n_head_k, n_tokens]
ggml_tensor * v_cur, // [n_embd_head_v, n_head_v, n_tokens]
ggml_tensor * kq_b, // [n_kv, n_tokens/n_stream, n_head_q, n_stream]
ggml_tensor * sinks, // [n_head_q]
float kq_scale,
int il) const;
//
// pooling
//
+2 -10
View File
@@ -5,7 +5,6 @@
#include "ggml.h"
#include <cstring>
#include <climits>
#include <cstdlib>
#include <stdexcept>
#include <cerrno>
@@ -32,9 +31,6 @@
#endif
#include <malloc.h>
#include <windows.h>
#ifndef PATH_MAX
#define PATH_MAX MAX_PATH
#endif
#include <io.h>
#endif
@@ -513,7 +509,7 @@ struct llama_file::impl {
if (errno == EINTR) {
continue;
}
if (errno == EFAULT || errno == EINVAL) {
if (errno == EFAULT || errno == EINVAL || errno == EIO) {
LLAMA_LOG_WARN("%s: file '%s' falling back to buffered I/O due to %s\n",
__func__, fname.c_str(), strerror(errno));
const size_t read_offset = tell();
@@ -522,7 +518,7 @@ struct llama_file::impl {
alignment = 1;
init_fp("rb");
seek(read_offset, SEEK_SET);
read_raw_unsafe(ptr, len);
read_raw_unsafe(static_cast<char *>(ptr) + bytes_read, to_read);
return;
}
throw std::runtime_error(format("read error: %s", strerror(errno)));
@@ -1140,7 +1136,3 @@ void llama_prefetch(llama_memory_ranges mr) {
GGML_UNUSED(mr);
#endif
}
size_t llama_path_max() {
return PATH_MAX;
}
-2
View File
@@ -88,5 +88,3 @@ using llama_memory_ranges = std::vector<llama_memory_range>;
// Prefetch the host pages covering these memory ranges.
void llama_prefetch(llama_memory_ranges mr);
size_t llama_path_max();
+2 -2
View File
@@ -84,7 +84,7 @@ const char * llama_ftype_name(llama_ftype ftype) {
static std::vector<std::string> llama_get_list_splits(const std::string & path, const int idx, const int n_split) {
std::vector<std::string> paths;
std::string split_prefix;
std::vector<char> buf(llama_path_max(), 0);
std::vector<char> buf(path.size() + 1, 0);
{
int ret = llama_split_prefix(buf.data(), buf.size(), path.c_str(), idx, n_split);
@@ -610,7 +610,7 @@ llama_model_loader::llama_model_loader(
throw std::runtime_error(format("illegal split file idx: %d (file: %s), model must be loaded with the first split", idx, fname.c_str()));
}
std::vector<char> prefix(llama_path_max(), 0);
std::vector<char> prefix(fname.size() + 1, 0);
if (paths.empty() || llama_split_prefix(prefix.data(), prefix.size(), fname.c_str(), idx, n_split)) {
auto backbone = llama_get_list_splits(fname, idx, n_split);
auto file_name = [](const std::string & path) {
+1 -1
View File
@@ -1266,7 +1266,7 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
GGML_ASSERT(ctx_outs[cur_split] && "Find uninitialized gguf_context");
std::string fname = fname_out;
if (params->keep_split) {
std::vector<char> split_path(llama_path_max(), 0);
std::vector<char> split_path(fname_out.size() + sizeof("-00000-of-00000.gguf"), 0);
llama_split_path(split_path.data(), split_path.size(), fname_out.c_str(), cur_split, n_split);
fname = std::string(split_path.data());
}
+290 -196
View File
@@ -2,111 +2,157 @@
#include "../llama-memory-hybrid-iswa.h"
void llama_model_inkling2::load_arch_hparams(llama_model_loader & ml) {
ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all);
ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
ml.get_key(LLM_KV_ATTENTION_QK_CLIP, hparams.f_qk_clip);
ml.get_key(LLM_KV_ATTN_RES_BLOCK_SIZE, hparams.attn_res_block_size);
ml.get_key(LLM_KV_SHORTCONV_L_CACHE, hparams.n_shortconv_l_cache);
// relative-position bias, and the log-scaled positions of the full-attention layers
ml.get_key(LLM_KV_ATTENTION_RELATIVE_LENGTH, hparams.n_relative);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_EXTENT, hparams.n_relative_extent);
ml.get_key(LLM_KV_ATTENTION_QK_CLIP, hparams.f_qk_clip);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_CLIP, hparams.f_relative_clip);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_CLIP, hparams.f_relative_clip);
ml.get_key(LLM_KV_ATTENTION_LOG_SCALE_FLOOR, hparams.n_log_scale_floor);
ml.get_key(LLM_KV_ATTENTION_LOG_SCALE_ALPHA, hparams.f_log_scale_alpha);
ml.get_key(LLM_KV_ATTN_RES_BLOCK_SIZE, hparams.attn_res_block_size);
ml.get_key(LLM_KV_SHORTCONV_L_CACHE, hparams.n_shortconv_l_cache);
ml.get_key(LLM_KV_VOCAB_SIZE, hparams.n_vocab_padded);
// the embedding and output tensors are padded past the vocabulary
ml.get_key(LLM_KV_VOCAB_SIZE, hparams.n_vocab_padded);
ml.get_key(LLM_KV_VOCAB_SIZE_UNPADDED, hparams.n_vocab_unpadded);
ml.get_key(LLM_KV_ACTIVATION_SITU_BETA, hparams.situ_beta);
// MoE
ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all);
ml.get_key(LLM_KV_ACTIVATION_SITU_BETA, hparams.situ_beta);
ml.get_key(LLM_KV_ACTIVATION_SITU_LINEAR_BETA, hparams.situ_linear_beta);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale);
ml.get_key(LLM_KV_FINAL_LOGIT_SOFTCAPPING, hparams.f_final_logit_softcapping);
ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale, false);
hparams.f_final_logit_softcapping = 0.0f;
ml.get_key(LLM_KV_FINAL_LOGIT_SOFTCAPPING, hparams.f_final_logit_softcapping, false);
// sliding window
ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
ml.get_arr(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, hparams.is_swa_impl);
uint32_t n_pattern = 0;
ml.get_arr_n(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, n_pattern);
if (n_pattern != hparams.n_layer() || !hparams.n_relative || !hparams.n_relative_extent || !hparams.n_swa ||
!hparams.attn_res_block_size || !hparams.n_log_scale_floor || hparams.n_shortconv_l_cache < 2 ||
hparams.n_shortconv_l_cache > 64 || hparams.n_relative_extent > INT32_MAX || hparams.n_swa > INT32_MAX ||
!hparams.n_vocab_unpadded || hparams.n_vocab_unpadded > hparams.n_vocab_padded) {
throw std::runtime_error("inkling2: invalid dimensions, vocabulary, or layer pattern");
}
for (float value : {hparams.f_norm_rms_eps, hparams.f_qk_clip, hparams.f_relative_clip, hparams.situ_beta,
hparams.situ_linear_beta, hparams.expert_weights_scale, hparams.f_final_logit_softcapping}) {
for (float value : {hparams.f_norm_rms_eps, hparams.f_qk_clip, hparams.f_relative_clip, hparams.situ_beta, hparams.situ_linear_beta}) {
if (!std::isfinite(value) || value <= 0.0f) {
throw std::runtime_error("inkling2: normalization, clipping and scaling parameters must be finite and positive");
}
}
if (!std::isfinite(hparams.f_log_scale_alpha) || hparams.f_log_scale_alpha < 0.0f) {
throw std::runtime_error("inkling2: invalid logarithmic scale");
}
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
hparams.expert_router_f32 = true;
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
hparams.expert_router_f32 = true;
hparams.expert_weights_norm_eps = 1e-20f;
// no RoPE, the relative-position bias carries the positions
hparams.rope_pattern.fill(false);
uint64_t max_channels = 0;
// the convolution state keeps K, V, the attention output and the FFN output of the widest layer
uint64_t n_channels_max = 0;
for (uint32_t il = 0; il < hparams.n_layer(); ++il) {
if (!hparams.n_head_kv(il) || hparams.n_head(il) % hparams.n_head_kv(il) ||
hparams.n_embd_head_k(il) != hparams.n_embd_head_v(il) || !hparams.n_expert_used(il) ||
hparams.n_expert_used(il) > hparams.n_expert || !hparams.n_ff_exp(il)) {
throw std::runtime_error("inkling2: invalid attention or expert dimensions");
}
max_channels = std::max<uint64_t>(max_channels, 2ull * hparams.n_embd_k_gqa(il) + 2ull * hparams.n_embd);
n_channels_max = std::max<uint64_t>(n_channels_max, 2ull * hparams.n_embd_k_gqa(il) + 2ull * hparams.n_embd);
}
if (max_channels * (hparams.n_shortconv_l_cache - 1) > UINT32_MAX) {
if (n_channels_max * (hparams.n_shortconv_l_cache - 1) > UINT32_MAX) {
throw std::runtime_error("inkling2: convolution state is too large");
}
hparams.n_sconv_channels = max_channels;
hparams.n_sconv_channels = n_channels_max;
type = LLM_TYPE_UNKNOWN;
}
void llama_model_inkling2::load_arch_tensors(llama_model_loader &) {
LLAMA_LOAD_LOCALS;
if (hparams.n_vocab_unpadded != vocab.n_tokens()) {
throw std::runtime_error("inkling2: tokenizer size must equal the unpadded vocabulary size");
}
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, hparams.n_vocab_padded}, 0);
const int64_t n_vocab_padded = hparams.n_vocab_padded;
const int64_t n_rel = hparams.n_relative;
const int64_t n_conv = hparams.n_shortconv_l_cache;
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab_padded}, 0);
tok_norm = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD_NORM, "weight", 0), {n_embd}, 0);
output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, hparams.n_vocab_padded}, 0);
output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
// output
output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab_padded}, 0);
output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
output_res_query = create_tensor(tn(LLM_TENSOR_OUTPUT_RES_QUERY, "weight"), {n_embd}, 0);
for (int i = 0; i < n_layer; ++i) {
auto & l = layers[i];
const int64_t nh = hparams.n_head(i);
const int64_t nk = hparams.n_head_kv(i);
const int64_t hd = hparams.n_embd_head_k(i);
const int64_t nr = hparams.n_relative;
const int64_t kernel = hparams.n_shortconv_l_cache;
const int64_t extent = hparams.is_swa(i) ? hparams.n_swa : hparams.n_relative_extent;
l.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), {n_embd, nh*hd}, 0);
l.wk = create_tensor(tn(LLM_TENSOR_ATTN_K, "weight", i), {n_embd, nk*hd}, 0);
l.wv = create_tensor(tn(LLM_TENSOR_ATTN_V, "weight", i), {n_embd, nk*hd}, 0);
l.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {nh*hd, n_embd}, 0);
l.wqkv_gate = create_tensor(tn(LLM_TENSOR_ATTN_GATE, "weight", i), {n_embd, nh*hd}, 0);
l.attn_rel = create_tensor(tn(LLM_TENSOR_ATTN_REL, "weight", i), {n_embd, nh*nr}, 0);
l.attn_rel_bias = create_tensor(tn(LLM_TENSOR_ATTN_REL_BIAS, "weight", i), {nr, extent}, 0);
l.attn_q_clip = create_tensor(tn(LLM_TENSOR_ATTN_Q_CLIP, "weight", i), {hd, nh}, 0);
l.attn_rel_clip = create_tensor(tn(LLM_TENSOR_ATTN_REL_CLIP, "weight", i), {nr, nh}, 0);
l.attn_sink_proj = create_tensor(tn(LLM_TENSOR_ATTN_SINK_PROJ, "weight", i), {n_embd, nh}, 0);
l.attn_k_conv = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV, "weight", i), {kernel, nk*hd}, 0);
l.attn_v_conv = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV, "weight", i), {kernel, nk*hd}, 0);
l.attn_post_conv = create_tensor(tn(LLM_TENSOR_ATTN_POST_CONV, "weight", i), {kernel, n_embd}, 0);
l.ffn_post_conv = create_tensor(tn(LLM_TENSOR_FFN_POST_CONV, "weight", i), {kernel, n_embd}, 0);
auto & layer = layers[i];
const int64_t n_head = hparams.n_head(i);
const int64_t n_head_kv = hparams.n_head_kv(i);
const int64_t n_embd_head = hparams.n_embd_head_k(i);
const int64_t n_ff_exp = hparams.n_ff_exp(i);
const int64_t n_extent = hparams.is_swa(i) ? hparams.n_swa : hparams.n_relative_extent;
// attention
layer.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), {n_embd, n_embd_head*n_head}, 0);
layer.wk = create_tensor(tn(LLM_TENSOR_ATTN_K, "weight", i), {n_embd, n_embd_head*n_head_kv}, 0);
layer.wv = create_tensor(tn(LLM_TENSOR_ATTN_V, "weight", i), {n_embd, n_embd_head*n_head_kv}, 0);
layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_embd_head*n_head, n_embd}, 0);
layer.wqkv_gate = create_tensor(tn(LLM_TENSOR_ATTN_GATE, "weight", i), {n_embd, n_embd_head*n_head}, 0);
layer.attn_q_clip = create_tensor(tn(LLM_TENSOR_ATTN_Q_CLIP, "weight", i), {n_embd_head, n_head}, 0);
layer.attn_sink_proj = create_tensor(tn(LLM_TENSOR_ATTN_SINK_PROJ, "weight", i), {n_embd, n_head}, 0);
// relative-position bias
layer.attn_rel = create_tensor(tn(LLM_TENSOR_ATTN_REL, "weight", i), {n_embd, n_rel*n_head}, 0);
layer.attn_rel_bias = create_tensor(tn(LLM_TENSOR_ATTN_REL_BIAS, "weight", i), {n_rel, n_extent}, 0);
layer.attn_rel_clip = create_tensor(tn(LLM_TENSOR_ATTN_REL_CLIP, "weight", i), {n_rel, n_head}, 0);
// short convolutions
layer.attn_k_conv = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV, "weight", i), {n_conv, n_embd_head*n_head_kv}, 0);
layer.attn_v_conv = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV, "weight", i), {n_conv, n_embd_head*n_head_kv}, 0);
layer.attn_post_conv = create_tensor(tn(LLM_TENSOR_ATTN_POST_CONV, "weight", i), {n_conv, n_embd}, 0);
layer.ffn_post_conv = create_tensor(tn(LLM_TENSOR_FFN_POST_CONV, "weight", i), {n_conv, n_embd}, 0);
if (hparams.is_swa(i)) {
l.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
} else {
l.attn_q_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_k_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_v_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_V_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_k_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV_PROJ, "weight", i), {n_embd, kernel*nk}, 0);
l.attn_v_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV_PROJ, "weight", i), {n_embd, kernel*nk}, 0);
// full-attention layers: separate Q, K and V input norms, and input-dependent taps for the K and V convolutions
layer.attn_q_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_k_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_v_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_V_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_k_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV_PROJ, "weight", i), {n_embd, n_conv*n_head_kv}, 0);
layer.attn_v_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV_PROJ, "weight", i), {n_embd, n_conv*n_head_kv}, 0);
}
// block attention residuals
if (i > 0) {
l.attn_res_query = create_tensor(tn(LLM_TENSOR_ATTN_RES_QUERY, "weight", i), {n_embd, hparams.is_swa(i) ? 1 : 3}, 0);
layer.attn_res_query = create_tensor(tn(LLM_TENSOR_ATTN_RES_QUERY, "weight", i), {n_embd, hparams.is_swa(i) ? 1 : 3}, 0);
}
l.ffn_res_query = create_tensor(tn(LLM_TENSOR_FFN_RES_QUERY, "weight", i), {n_embd}, 0);
l.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
l.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0);
l.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, TENSOR_NOT_REQUIRED);
l.ffn_gate_up_exps = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "weight", i), {n_embd, 2*hparams.n_ff_exp(i), n_expert}, 0);
l.ffn_down_exps = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {hparams.n_ff_exp(i), n_embd, n_expert}, 0);
layer.ffn_res_query = create_tensor(tn(LLM_TENSOR_FFN_RES_QUERY, "weight", i), {n_embd}, 0);
// MoE
layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
layer.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0);
layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, TENSOR_NOT_REQUIRED);
layer.ffn_gate_up_exps = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "weight", i), {n_embd, 2*n_ff_exp, n_expert}, 0);
layer.ffn_down_exps = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
}
}
@@ -114,202 +160,250 @@ std::unique_ptr<llm_graph_context> llama_model_inkling2::build_arch_graph(const
return std::make_unique<graph>(params);
}
class llm_graph_input_inkling2_pos : public llm_graph_input_i {
public:
const llama_kv_cache_context * cache[2];
ggml_tensor * distance[2];
uint32_t extent[2];
bool can_reuse(const llm_graph_params & params) override {
const auto * memory = static_cast<const llama_memory_hybrid_iswa_context *>(params.mctx);
cache[0] = memory->get_attn()->get_base();
cache[1] = memory->get_attn()->get_swa();
for (int i = 0; i < 2; ++i) {
if (distance[i]->ne[0] != cache[i]->get_n_kv() || distance[i]->ne[1] != params.ubatch.n_tokens) {
return false;
}
}
return true;
}
void set_input(const llama_ubatch * ubatch) override {
for (int i = 0; i < 2; ++i) {
if (distance[i]->buffer) {
cache[i]->set_input_relative_pos(distance[i], ubatch, extent[i]);
}
}
}
};
// scale each row down to an L2 norm of at most threshold, after the optional per-channel scale
ggml_tensor * llama_model_inkling2::graph::clip(ggml_tensor * cur, ggml_tensor * scale, float threshold) {
if (scale) {
cur = ggml_mul(ctx0, cur, scale);
}
auto * norm = ggml_sqrt(ctx0, ggml_sum_rows(ctx0, ggml_sqr(ctx0, cur)));
ggml_tensor * norm = ggml_sqrt(ctx0, ggml_sum_rows(ctx0, ggml_sqr(ctx0, cur)));
norm = ggml_scale_bias(ctx0, norm, 1.0f, 1e-6f);
auto * factor = ggml_div(ctx0, ggml_fill(ctx0, norm, threshold), norm);
ggml_tensor * factor = ggml_div(ctx0, ggml_fill(ctx0, norm, threshold), norm);
return ggml_mul(ctx0, cur, ggml_clamp(ctx0, factor, 0.0f, 1.0f));
}
// attention over the depth: mix the earlier block outputs and cur with learned queries
ggml_tensor * llama_model_inkling2::graph::mix(const std::vector<ggml_tensor *> & blocks, ggml_tensor * cur, ggml_tensor * query) {
auto * states = ggml_reshape_3d(ctx0, cur, n_embd, 1, n_tokens);
for (auto * block : blocks) {
ggml_tensor * states = ggml_reshape_3d(ctx0, cur, n_embd, 1, n_tokens);
for (ggml_tensor * block : blocks) {
states = ggml_concat(ctx0, ggml_reshape_3d(ctx0, block, n_embd, 1, n_tokens), states, 1);
}
states = ggml_rms_norm(ctx0, states, 1e-6f);
auto * queries = ggml_repeat(ctx0, query, ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, n_embd, query->ne[1], n_tokens));
auto * scores = ggml_mul_mat(ctx0, states, queries);
ggml_tensor * queries = ggml_repeat(ctx0, query, ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, n_embd, query->ne[1], n_tokens));
ggml_tensor * scores = ggml_mul_mat(ctx0, states, queries);
scores = ggml_soft_max_ext(ctx0, scores, nullptr, 1.0f/sqrtf(n_embd), 0.0f);
return ggml_mul_mat(ctx0, ggml_cont(ctx0, ggml_transpose(ctx0, states)), scores);
}
ggml_tensor * llama_model_inkling2::graph::convolve(ggml_tensor * cur, ggml_tensor * weight, ggml_tensor * dynamic,
ggml_tensor * state, ggml_tensor * history, llm_graph_input_rs * inp, int64_t offset) {
const int64_t kernel = hparams.n_shortconv_l_cache;
const int64_t channels = cur->ne[0];
const int64_t nt = ubatch.n_seq_tokens;
const int64_t ns = ubatch.n_seqs;
auto * past = ggml_view_3d(ctx0, history, kernel - 1, channels, ns, history->nb[1], history->nb[2], offset*history->nb[1]);
auto * x = ggml_transpose(ctx0, ggml_reshape_3d(ctx0, cur, channels, nt, ns));
// causal short convolution with a residual; dynamic adds input-dependent taps per head
// the last n_conv - 1 inputs of each sequence live in the recurrent state at channel offset `offset`
ggml_tensor * llama_model_inkling2::graph::convolve(
ggml_tensor * cur,
ggml_tensor * weight,
ggml_tensor * dynamic,
ggml_tensor * state,
ggml_tensor * history,
llm_graph_input_rs * inp,
int64_t offset) {
const int64_t n_conv = hparams.n_shortconv_l_cache;
const int64_t n_channels = cur->ne[0];
const int64_t n_seq_tokens = ubatch.n_seq_tokens;
const int64_t n_seqs = ubatch.n_seqs;
ggml_tensor * past = ggml_view_3d(ctx0, history, n_conv - 1, n_channels, n_seqs, history->nb[1], history->nb[2], offset*history->nb[1]);
ggml_tensor * x = ggml_transpose(ctx0, ggml_reshape_3d(ctx0, cur, n_channels, n_seq_tokens, n_seqs));
x = ggml_concat(ctx0, past, x, 0);
build_conv_state_store(inp, state, x, kernel - 1, offset*(kernel - 1)*ggml_element_size(state));
auto * out = ggml_ssm_conv(ctx0, x, weight);
out = ggml_reshape_2d(ctx0, out, channels, n_tokens);
build_conv_state_store(inp, state, x, n_conv - 1, offset*(n_conv - 1)*ggml_element_size(state));
ggml_tensor * out = ggml_ssm_conv(ctx0, x, weight);
out = ggml_reshape_2d(ctx0, out, n_channels, n_tokens);
if (dynamic) {
const int64_t heads = dynamic->ne[0]/kernel;
dynamic = ggml_reshape_3d(ctx0, dynamic, kernel, heads, n_tokens);
for (int64_t lag = 0; lag < kernel; ++lag) {
auto * tap = ggml_view_3d(ctx0, dynamic, 1, heads, n_tokens, dynamic->nb[1], dynamic->nb[2], lag*dynamic->nb[0]);
auto * window = ggml_view_3d(ctx0, x, nt, channels, ns, x->nb[1], x->nb[2], (kernel - 1 - lag)*x->nb[0]);
const int64_t n_heads = dynamic->ne[0]/n_conv;
dynamic = ggml_reshape_3d(ctx0, dynamic, n_conv, n_heads, n_tokens);
for (int64_t lag = 0; lag < n_conv; ++lag) {
ggml_tensor * tap = ggml_view_3d(ctx0, dynamic, 1, n_heads, n_tokens, dynamic->nb[1], dynamic->nb[2], lag*dynamic->nb[0]);
ggml_tensor * window = ggml_view_3d(ctx0, x, n_seq_tokens, n_channels, n_seqs, x->nb[1], x->nb[2], (n_conv - 1 - lag)*x->nb[0]);
window = ggml_cont(ctx0, ggml_transpose(ctx0, window));
window = ggml_reshape_3d(ctx0, window, channels/heads, heads, n_tokens);
auto * term = ggml_reshape_2d(ctx0, ggml_mul(ctx0, window, tap), channels, n_tokens);
window = ggml_reshape_3d(ctx0, window, n_channels/n_heads, n_heads, n_tokens);
ggml_tensor * term = ggml_reshape_2d(ctx0, ggml_mul(ctx0, window, tap), n_channels, n_tokens);
out = ggml_add(ctx0, out, term);
}
}
return ggml_scale(ctx0, ggml_add(ctx0, cur, out), 1.0f/sqrtf((dynamic ? 2 : 1)*kernel + 1));
return ggml_scale(ctx0, ggml_add(ctx0, cur, out), 1.0f/sqrtf((dynamic ? 2 : 1)*n_conv + 1));
}
llama_model_inkling2::graph::graph(const llm_graph_params & params) : llm_graph_context(params) {
auto * inp = build_inp_mem_hybrid_iswa();
auto * inp = build_inp_mem_hybrid_iswa_rel(hparams.n_relative_extent, hparams.n_swa);
auto * attn = inp->get_attn();
auto * recr = inp->get_recr();
auto * positions = build_inp_pos();
auto * out_ids = build_inp_out_ids();
auto rel = std::make_unique<llm_graph_input_inkling2_pos>();
rel->cache[0] = attn->mctx->get_base();
rel->cache[1] = attn->mctx->get_swa();
rel->extent[0] = hparams.n_relative_extent;
rel->extent[1] = hparams.n_swa;
ggml_tensor * distances[2];
for (int i = 0; i < 2; ++i) {
distances[i] = rel->distance[i] = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, rel->cache[i]->get_n_kv(), n_tokens);
ggml_set_input(distances[i]);
}
res->add_input(std::move(rel));
auto * tau = ggml_scale_bias(ctx0, ggml_cast(ctx0, positions, GGML_TYPE_F32), 1.0f, 1.0f);
ggml_tensor * inp_pos = build_inp_pos();
ggml_tensor * inp_out_ids = build_inp_out_ids();
// log-scaled positions: 1 + alpha*log(max(1, (pos + 1)/floor)), multiplies q and r of the full-attention layers
ggml_tensor * tau = ggml_scale_bias(ctx0, ggml_cast(ctx0, inp_pos, GGML_TYPE_F32), 1.0f, 1.0f);
tau = ggml_log(ctx0, ggml_clamp(ctx0, ggml_scale(ctx0, tau, 1.0f/hparams.n_log_scale_floor), 1.0f, INFINITY));
tau = ggml_reshape_3d(ctx0, ggml_scale_bias(ctx0, tau, hparams.f_log_scale_alpha, 1.0f), 1, 1, n_tokens);
auto * cur = build_inp_embd(model.tok_embd);
ggml_tensor * cur = build_inp_embd(model.tok_embd);
// important: do not normalize raw embeddings input (i.e. encoded image embeddings)
if (ubatch.token) {
cur = build_norm(cur, model.tok_norm, nullptr, LLM_NORM_RMS, -1);
}
std::vector<ggml_tensor *> blocks;
for (int il = 0; il < n_layer; ++il) {
const auto & l = model.layers[il];
const bool swa = hparams.is_swa(il);
const int64_t nh = hparams.n_head(il);
const int64_t nk = hparams.n_head_kv(il);
const int64_t hd = hparams.n_embd_head_k(il);
const int64_t kv = nk*hd;
auto * state = recr->mctx->get_r_l(il);
auto * history = build_rs(recr, state, hparams.n_embd_r(), ubatch.n_seqs);
const auto & layer = model.layers[il];
const bool is_swa = hparams.is_swa(il);
const int64_t n_head = hparams.n_head(il);
const int64_t n_head_kv = hparams.n_head_kv(il);
const int64_t n_embd_head = hparams.n_embd_head_k(il);
const int64_t n_embd_k_gqa = n_embd_head*n_head_kv;
ggml_tensor * state = recr->mctx->get_r_l(il);
ggml_tensor * history = build_rs(recr, state, hparams.n_embd_r(), ubatch.n_seqs);
history = ggml_reshape_3d(ctx0, history, hparams.n_shortconv_l_cache - 1, hparams.n_sconv_channels, ubatch.n_seqs);
auto * residual = cur;
ggml_tensor * inpSA = cur;
// attention inputs: SWA layers mix one, full-attention layers one each for Q, K and V
ggml_tensor * inputs[3] = {cur, cur, cur};
if (l.attn_res_query) {
auto * mixed = mix(blocks, cur, l.attn_res_query);
if (layer.attn_res_query) {
ggml_tensor * mixed = mix(blocks, cur, layer.attn_res_query);
for (int j = 0; j < 3; ++j) {
inputs[j] = ggml_cont(ctx0, ggml_view_2d(ctx0, mixed, n_embd, n_tokens, mixed->nb[2], (swa ? 0 : j)*mixed->nb[1]));
inputs[j] = ggml_cont(ctx0, ggml_view_2d(ctx0, mixed, n_embd, n_tokens, mixed->nb[2], (is_swa ? 0 : j)*mixed->nb[1]));
}
}
if (swa) {
inputs[0] = build_norm(inputs[0], l.attn_norm, nullptr, LLM_NORM_RMS, il);
if (is_swa) {
inputs[0] = build_norm(inputs[0], layer.attn_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = inputs[2] = inputs[0];
} else {
inputs[0] = build_norm(inputs[0], l.attn_q_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = build_norm(inputs[1], l.attn_k_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[2] = build_norm(inputs[2], l.attn_v_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[0] = build_norm(inputs[0], layer.attn_q_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = build_norm(inputs[1], layer.attn_k_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[2] = build_norm(inputs[2], layer.attn_v_inp_norm, nullptr, LLM_NORM_RMS, il);
}
auto * q = build_lora_mm(l.wq, inputs[0]);
auto * k = convolve(build_lora_mm(l.wk, inputs[1]), l.attn_k_conv,
swa ? nullptr : build_lora_mm(l.attn_k_conv_proj, inputs[1]), state, history, recr, 0);
auto * v = convolve(build_lora_mm(l.wv, inputs[2]), l.attn_v_conv,
swa ? nullptr : build_lora_mm(l.attn_v_conv_proj, inputs[2]), state, history, recr, kv);
auto * r = build_lora_mm(l.attn_rel, inputs[0]);
q = clip(ggml_reshape_3d(ctx0, q, hd, nh, n_tokens), l.attn_q_clip, hparams.f_qk_clip);
k = clip(ggml_reshape_3d(ctx0, k, hd, nk, n_tokens), nullptr, hparams.f_qk_clip);
v = ggml_rms_norm(ctx0, ggml_reshape_3d(ctx0, v, hd, nk, n_tokens), hparams.f_norm_rms_eps);
r = clip(ggml_reshape_3d(ctx0, r, hparams.n_relative, nh, n_tokens), l.attn_rel_clip, hparams.f_relative_clip);
if (!swa) {
q = ggml_mul(ctx0, q, tau);
r = ggml_mul(ctx0, r, tau);
// self-attention
{
ggml_tensor * Qcur = build_lora_mm(layer.wq, inputs[0]);
ggml_tensor * Kcur = convolve(build_lora_mm(layer.wk, inputs[1]), layer.attn_k_conv,
is_swa ? nullptr : build_lora_mm(layer.attn_k_conv_proj, inputs[1]), state, history, recr, 0);
ggml_tensor * Vcur = convolve(build_lora_mm(layer.wv, inputs[2]), layer.attn_v_conv,
is_swa ? nullptr : build_lora_mm(layer.attn_v_conv_proj, inputs[2]), state, history, recr, n_embd_k_gqa);
ggml_tensor * Rcur = build_lora_mm(layer.attn_rel, inputs[0]);
Qcur = clip(ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head, n_tokens), layer.attn_q_clip, hparams.f_qk_clip);
Kcur = clip(ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens), nullptr, hparams.f_qk_clip);
Vcur = ggml_rms_norm(ctx0, ggml_reshape_3d(ctx0, Vcur, n_embd_head, n_head_kv, n_tokens), hparams.f_norm_rms_eps);
Rcur = clip(ggml_reshape_3d(ctx0, Rcur, hparams.n_relative, n_head, n_tokens), layer.attn_rel_clip, hparams.f_relative_clip);
if (!is_swa) {
Qcur = ggml_mul(ctx0, Qcur, tau);
Rcur = ggml_mul(ctx0, Rcur, tau);
}
cb(Qcur, "Qcur", il);
cb(Kcur, "Kcur", il);
cb(Vcur, "Vcur", il);
cb(Rcur, "Rcur", il);
ggml_tensor * rel_pos = is_swa ? inp->self_rel_pos_swa : inp->self_rel_pos;
const int64_t n_stream = (is_swa ? attn->self_attn_mask_swa.raw : attn->self_attn_mask.raw)->ne[3];
const int64_t n_kv = rel_pos->ne[0];
ggml_tensor * sinks = build_lora_mm(layer.attn_sink_proj, inputs[0]);
// project r on the bias table first, then gather by distance: only the gathered bias is n_kv x n_tokens x n_head
ggml_tensor * table = ggml_mul_mat(ctx0, ggml_pad(ctx0, layer.attn_rel_bias, 0, 1, 0, 0), Rcur);
// the sink logit t is per query: exp(s - t)/(1 + sum(exp(s - t))) = exp(s)/(exp(t) + sum(exp(s))), with a zero sink
table = ggml_sub(ctx0, table, ggml_reshape_3d(ctx0, sinks, 1, n_head, n_tokens));
table = ggml_cont(ctx0, ggml_permute(ctx0, table, 1, 0, 2, 3));
ggml_tensor * kq_b = ggml_get_rows(ctx0, table, rel_pos);
kq_b = ggml_reshape_4d(ctx0, kq_b, n_head, n_kv, n_tokens/n_stream, n_stream);
kq_b = ggml_cont(ctx0, ggml_permute(ctx0, kq_b, 2, 0, 1, 3));
cb(kq_b, "kq_b", il);
sinks = ggml_fill(ctx0, ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, n_head), 0.0f);
cur = build_attn(inp,
nullptr, nullptr, nullptr,
Qcur, Kcur, Vcur, kq_b, sinks, 1.0f/sqrtf(n_embd_head), il);
cur = ggml_mul(ctx0, cur, ggml_sigmoid(ctx0, build_lora_mm(layer.wqkv_gate, inputs[0])));
cur = convolve(build_lora_mm(layer.wo, cur), layer.attn_post_conv, nullptr, state, history, recr, 2*n_embd_k_gqa);
cb(cur, "attn_out", il);
}
auto * mask = swa ? attn->self_attn_mask_swa.raw : attn->self_attn_mask.raw;
const int64_t streams = mask->ne[3];
const int64_t n_kv = distances[swa]->ne[0];
auto * sinks = build_lora_mm(l.attn_sink_proj, inputs[0]);
// project r on the bias table first, then gather by distance: only the gathered bias is n_kv x n_tokens x nh
auto * table = ggml_mul_mat(ctx0, ggml_pad(ctx0, l.attn_rel_bias, 0, 1, 0, 0), r);
if (cparams.flash_attn) {
// exp(s - t)/(1 + sum(exp(s - t))) = exp(s)/(exp(t) + sum(exp(s))).
table = ggml_sub(ctx0, table, ggml_reshape_3d(ctx0, sinks, 1, nh, n_tokens));
} else {
// build_attn scales the bias along with QK; FA adds the mask after scaling QK.
table = ggml_scale(ctx0, table, sqrtf(hd));
}
table = ggml_cont(ctx0, ggml_permute(ctx0, table, 1, 0, 2, 3));
auto * bias = ggml_get_rows(ctx0, table, distances[swa]);
bias = ggml_cont(ctx0, ggml_permute(ctx0, ggml_reshape_4d(ctx0, bias, nh, n_kv, n_tokens/streams, streams), 2, 0, 1, 3));
ggml_tensor * fa_mask = nullptr;
if (cparams.flash_attn) {
fa_mask = ggml_cast(ctx0, ggml_add(ctx0, bias, ggml_cast(ctx0, mask, GGML_TYPE_F32)), GGML_TYPE_F16);
sinks = ggml_fill(ctx0, ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, nh), 0.0f);
bias = nullptr;
} else {
sinks = ggml_reshape_4d(ctx0, ggml_cont(ctx0, ggml_permute(ctx0,
ggml_reshape_3d(ctx0, sinks, nh, n_tokens/streams, streams), 1, 0, 2, 3)), 1, n_tokens/streams, nh, streams);
}
cur = build_attn(attn, nullptr, nullptr, nullptr, q, k, v, bias, sinks, nullptr, 1.0f/sqrtf(hd), il, fa_mask);
cur = ggml_mul(ctx0, cur, ggml_sigmoid(ctx0, build_lora_mm(l.wqkv_gate, inputs[0])));
cur = convolve(build_lora_mm(l.wo, cur), l.attn_post_conv, nullptr, state, history, recr, 2*kv);
// the first layer of each block starts a new residual stream, the earlier one joins the mixed blocks
if (il % hparams.attn_res_block_size == 0) {
blocks.push_back(residual);
blocks.push_back(inpSA);
} else {
cur = ggml_add(ctx0, residual, cur);
cur = ggml_add(ctx0, inpSA, cur);
}
residual = cur;
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, l.ffn_res_query), n_embd, n_tokens);
cur = build_norm(cur, l.ffn_norm, nullptr, LLM_NORM_RMS, il);
cur = build_moe_ffn(cur, l.ffn_gate_inp, nullptr, nullptr, l.ffn_down_exps, l.ffn_exp_probs_b,
hparams.n_expert, hparams.n_expert_used(il), LLM_FFN_SITU, true, hparams.expert_weights_scale,
LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID, il, nullptr, l.ffn_gate_up_exps);
cur = convolve(cur, l.ffn_post_conv, nullptr, state, history, recr, 2*kv + n_embd);
cur = ggml_add(ctx0, residual, cur);
ggml_tensor * ffn_inp = cur;
cb(ffn_inp, "ffn_inp", il);
// MoE
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, layer.ffn_res_query), n_embd, n_tokens);
cur = build_norm(cur, layer.ffn_norm, nullptr, LLM_NORM_RMS, il);
cur = build_moe_ffn(cur,
layer.ffn_gate_inp,
nullptr,
nullptr,
layer.ffn_down_exps,
layer.ffn_exp_probs_b,
hparams.n_expert,
hparams.n_expert_used(il),
LLM_FFN_SITU, true,
hparams.expert_weights_scale,
LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID,
il,
nullptr,
layer.ffn_gate_up_exps);
cur = convolve(cur, layer.ffn_post_conv, nullptr, state, history, recr, 2*n_embd_k_gqa + n_embd);
cb(cur, "ffn_out", il);
cur = ggml_add(ctx0, ffn_inp, cur);
cb(cur, "l_out", il);
}
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, model.output_res_query), n_embd, n_tokens);
cur = build_norm(cur, model.output_norm, nullptr, LLM_NORM_RMS, -1);
if (out_ids) {
cur = ggml_get_rows(ctx0, cur, out_ids);
if (inp_out_ids) {
cur = ggml_get_rows(ctx0, cur, inp_out_ids);
}
cb(cur, "result_norm", -1);
res->t_embd = cur;
// lm_head
cur = build_lora_mm(model.output, cur);
ggml_prec_set_acc(cur, GGML_PREC_F32);
const float cap = hparams.f_final_logit_softcapping;
cur = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/cap)), cap);
if (hparams.f_final_logit_softcapping != 0.0f) {
const float cap = hparams.f_final_logit_softcapping;
cur = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/cap)), cap);
}
// drop the padded vocabulary
cur = ggml_cont(ctx0, ggml_view_2d(ctx0, cur, hparams.n_vocab_unpadded, cur->ne[1], cur->nb[1], 0));
res->t_logits = cur;
cb(cur, "result_output", -1);
res->t_logits = cur;
}
+8 -1
View File
@@ -2480,7 +2480,14 @@ struct llama_model_inkling2 : public llama_model_base {
graph(const llm_graph_params & params);
ggml_tensor * mix(const std::vector<ggml_tensor *> & blocks, ggml_tensor * cur, ggml_tensor * query);
ggml_tensor * clip(ggml_tensor * cur, ggml_tensor * scale, float threshold);
ggml_tensor * convolve(ggml_tensor * cur, ggml_tensor * weight, ggml_tensor * dynamic, ggml_tensor * state, ggml_tensor * history, llm_graph_input_rs * inp, int64_t offset);
ggml_tensor * convolve(
ggml_tensor * cur,
ggml_tensor * weight,
ggml_tensor * dynamic,
ggml_tensor * state,
ggml_tensor * history,
llm_graph_input_rs * inp,
int64_t offset);
};
std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
+2
View File
@@ -85,6 +85,8 @@ llama_model_qwen3vl::graph::graph(const llm_graph_params & params) : llm_graph_c
ggml_tensor * inp_out_ids = build_inp_out_ids();
for (int il = 0; il < n_layer; ++il) {
res->t_layer_inp[il] = inpL;
ggml_tensor * inpSA = inpL;
// norm
+2
View File
@@ -367,6 +367,8 @@ endif()
llama_build(test-backend-ops.cpp LINK ggml Threads::Threads)
llama_build_and_test(test-backend-tensor.cpp LINK ggml)
llama_build_and_test(test-sleep.cpp LINK ggml)
llama_build_and_test(test-backend-sched.cpp LINK ggml)
if (BUILD_SHARED_LIBS)
llama_build_and_test(test-llama-mmap.cpp ${PROJECT_SOURCE_DIR}/src/llama-impl.cpp ${PROJECT_SOURCE_DIR}/src/llama-mmap.cpp LABEL behavior)
+40
View File
@@ -9657,6 +9657,33 @@ struct test_fill : public test_case {
}
};
// GGML_OP_SLEEP (for scheduler testing only)
struct test_sleep : public test_case {
const ggml_type type;
const std::array<int64_t, 4> ne;
const int32_t us;
std::string vars() override { return VARS_TO_STR3(type, ne, us); }
// the op only delays, the output aliases the input and must match it exactly
double max_nmse_err() override { return 0.0; }
test_sleep(int32_t us = 100, ggml_type type = GGML_TYPE_F32,
std::array<int64_t, 4> ne = { 10, 10, 4, 3 })
: type(type), ne(ne), us(us) {}
ggml_tensor * build_graph(ggml_context * ctx) override {
ggml_tensor * a = ggml_new_tensor_4d(ctx, type, ne[0], ne[1], ne[2], ne[3]);
ggml_set_name(a, "a");
ggml_tensor * out = ggml_sleep(ctx, a, us);
ggml_set_name(out, "out");
return out;
}
};
// GGML_OP_SOLVE_TRI
struct test_solve_tri : public test_case {
const ggml_type type;
@@ -11917,6 +11944,9 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
// > 2^31-element dst (8.9 GB), about 45 GB peak memory. Too large to be enabled by default.
// kernel_mul_mm dst offset of the last batch is past INT32_MAX: rows 0-31 use full tiles, rows 32-39 use edge tiles.
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 1048576, 40, 64, {1, 1}, {53, 1}));
// CUDA mul_mat_q: > 2^31-element dst (8.9 GB), the dst offset of column tiles starting past column 8648 does not fit in 32 bits
// (same shape class as a 248k-vocab output head with a 16k ubatch of outputs)
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q8_0, GGML_TYPE_F32, 248320, 8960, 32, {1, 1}, {1, 1}));
// kernel_mul_mm_id: the index of tokens 32768-33791 does not fit in 16 bits (about 0.3 GB peak memory).
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_F32, GGML_TYPE_F32, 4, 2, false, 64, 33792, 64));
@@ -12262,6 +12292,12 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 576, 512, 576, {1,1}, {1,1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 1, 2048, 8192, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 128, 128, 4352, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 64, 192, 17664, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 128, 128, 4096, {1, 1}, {2, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_K, GGML_TYPE_F32, 128, 256, 4096, {1, 1}, {1, 1}));
for (ggml_type type_a : all_types) {
test_cases.emplace_back(new test_mul_mat(type_a, GGML_TYPE_F32, 1, 64, 256, {1, 1}, {1, 1}));
}
@@ -12987,6 +13023,10 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_fill(-152.0f, GGML_TYPE_F16, { 800, 600, 4, 4 }));
test_cases.emplace_back(new test_fill(3.5f, GGML_TYPE_F16, { 2048, 512, 2, 2 }));
test_cases.emplace_back(new test_sleep(0));
test_cases.emplace_back(new test_sleep(10, GGML_TYPE_F16, { 128, 4, 2, 2 }));
test_cases.emplace_back(new test_sleep(10, GGML_TYPE_F32, { 128, 4, 2, 2 }));
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16}) {
test_cases.emplace_back(new test_diag(type));
test_cases.emplace_back(new test_diag(type, { 79, 1, 19, 13 }));
File diff suppressed because it is too large Load Diff
+34
View File
@@ -170,6 +170,40 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
}
}
{
std::vector<ggml_tensor *> batch;
ggml_backend_buffer_ptr scatter_buf = alloc_tensors(ctx.get(), buft, { 64 }, batch);
if (!scatter_buf) {
return false;
}
batch.resize(128, batch[0]);
uint8_t input[32], expected[256], out[256];
for (size_t i = 0; i < sizeof(input); i++) {
input[i] = (uint8_t) (i + 1);
}
std::vector<size_t> offsets(batch.size()), sizes(batch.size(), 1), data_offsets(batch.size());
for (bool overlap : { false, true }) {
memset(expected, 0, sizeof(expected));
ggml_backend_tensor_set(batch[0], expected, 0, sizeof(expected));
for (size_t i = 0; i < batch.size(); i++) {
offsets[i] = 2*((i*5) % batch.size()) + 1;
data_offsets[i] = i % sizeof(input);
}
// Repeated sources are valid; overlapping destinations must retain copy order.
if (overlap) {
offsets.back() = offsets.front();
}
for (size_t i = 0; i < batch.size(); i++) {
expected[offsets[i]] = input[data_offsets[i]];
}
ggml_backend_tensor_set_n(batch.data(), input, offsets.data(), sizes.data(), data_offsets.data(), batch.size());
ggml_backend_tensor_get(batch[0], out, 0, sizeof(out));
if (memcmp(out, expected, sizeof(out)) != 0) {
return false;
}
}
}
return true;
}
+154
View File
@@ -0,0 +1,154 @@
// Tests for GGML_OP_SLEEP, on every backend that supports it. Three properties are checked:
// - the op only delays: input = output
// - the op does not return early: a chain of n sleeps must take at least n*us
// Only a lower bound is asserted for the duration. A busy-wait cannot finish early, so that bound is
// deterministic, whereas an upper bound would be flaky on a loaded machine.
// Known limitation: Vulkan does not specify the tick rate of the shader clock, see ggml_vk_sleep. A driver
// whose clock runs slower than assumed could undershoot the request and fail this bound.
#include "ggml.h"
#include "ggml-alloc.h"
#include "ggml-backend.h"
#include "ggml-cpp.h"
#include <cinttypes>
#include <cstdio>
#include <cstring>
#include <random>
#include <vector>
enum test_result {
TEST_OK,
TEST_FAIL,
TEST_SKIP,
};
static test_result test_sleep(ggml_backend_t backend, ggml_type type, int64_t ne, int32_t us, int n_nodes) {
const size_t graph_size = n_nodes + 16;
ggml_init_params params_static = {
/*.mem_size =*/ ggml_tensor_overhead(),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ggml_context_ptr ctx_static(ggml_init(params_static));
ggml_tensor * x = ggml_new_tensor_1d(ctx_static.get(), type, ne);
ggml_set_name(x, "x");
ggml_set_input(x);
ggml_init_params params_compute = {
/*.mem_size =*/ n_nodes*ggml_tensor_overhead() + ggml_graph_overhead_custom(graph_size, false),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ggml_context_ptr ctx_compute(ggml_init(params_compute));
ggml_cgraph * gf = ggml_new_graph_custom(ctx_compute.get(), graph_size, false);
ggml_tensor * out = x;
for (int i = 0; i < n_nodes; i++) {
out = ggml_sleep(ctx_compute.get(), out, us);
}
ggml_set_name(out, "out");
ggml_build_forward_expand(gf, out);
// the backends log to stdout while computing, so the result line is only printed once it is complete
char desc[128];
snprintf(desc, sizeof(desc), " %-6s %-4s ne=%-9" PRId64 " us=%-7d n_nodes=%d ",
ggml_backend_name(backend), ggml_type_name(type), ne, us, n_nodes);
test_result result = TEST_OK;
if (!ggml_backend_supports_op(backend, out)) {
printf("%snot supported\n", desc);
result = TEST_SKIP;
} else {
ggml_backend_buffer_ptr buf_static(ggml_backend_alloc_ctx_tensors(ctx_static.get(), backend));
// random bytes rather than random floats, so that bit patterns a value-wise copy could
// mangle, such as NaNs and denormals, are covered as well
std::vector<uint8_t> data_in(ggml_nbytes(x));
std::mt19937 rng(1234);
for (size_t i = 0; i < data_in.size(); i++) {
data_in[i] = rng() & 0xFF;
}
ggml_backend_tensor_set(x, data_in.data(), 0, data_in.size());
ggml_gallocr_ptr galloc(ggml_gallocr_new(ggml_backend_get_default_buffer_type(backend)));
ggml_gallocr_alloc_graph(galloc.get(), gf);
// warm up, so that one-time costs such as lazy CUDA module loading are not timed
ggml_backend_graph_compute(backend, gf);
const int64_t t_start_us = ggml_time_us();
const ggml_status status = ggml_backend_graph_compute(backend, gf);
const int64_t t_us = ggml_time_us() - t_start_us;
std::vector<uint8_t> data_out(ggml_nbytes(out));
ggml_backend_tensor_get(out, data_out.data(), 0, data_out.size());
const int64_t t_min_us = (int64_t) n_nodes * us;
const bool ok_status = status == GGML_STATUS_SUCCESS;
const bool ok_alias = out->data == x->data;
const bool ok_data = memcmp(data_in.data(), data_out.data(), data_in.size()) == 0;
const bool ok_time = t_us >= t_min_us;
printf("%selapsed=%7" PRId64 " us (>= %6" PRId64 ") alias: %-4s data: %-4s time: %s\n",
desc, t_us, t_min_us, ok_alias ? "OK" : "FAIL", ok_data ? "OK" : "FAIL", ok_time ? "OK" : "FAIL");
if (!ok_status) {
printf(" compute failed: %s\n", ggml_status_to_string(status));
}
result = ok_status && ok_alias && ok_data && ok_time ? TEST_OK : TEST_FAIL;
}
return result;
}
int main() {
ggml_time_init();
ggml_backend_load_all();
int n_ok = 0;
int n_fail = 0;
int n_skip = 0;
for (size_t i = 0; i < ggml_backend_dev_count(); i++) {
ggml_backend_dev_t dev = ggml_backend_dev_get(i);
ggml_backend_ptr backend(ggml_backend_dev_init(dev, nullptr));
if (backend == nullptr) {
printf(" failed to initialize %s\n", ggml_backend_dev_name(dev));
n_fail++;
continue;
}
// a zero duration must still be a valid no-op, longer ones are checked against the clock,
// and the last case verifies that a chain of sleeps accumulates instead of collapsing into one
const struct {
ggml_type type;
int64_t ne;
int32_t us;
int n_nodes;
} cases[] = {
{ GGML_TYPE_F32, 1, 0, 1 },
{ GGML_TYPE_F32, 4096, 10000, 1 },
{ GGML_TYPE_F16, 1024, 10000, 1 },
{ GGML_TYPE_F32, 1 << 20, 5000, 4 },
};
for (const auto & c : cases) {
switch (test_sleep(backend.get(), c.type, c.ne, c.us, c.n_nodes)) {
case TEST_OK: n_ok++; break;
case TEST_FAIL: n_fail++; break;
case TEST_SKIP: n_skip++; break;
}
}
}
printf("%d passed, %d failed, %d skipped\n", n_ok, n_fail, n_skip);
return n_fail > 0;
}
+4
View File
@@ -353,6 +353,8 @@ struct server_slot {
return false;
}
const int64_t t_start = ggml_time_us();
const size_t cur_size_tgt = llama_state_seq_get_size_ext(ctx_tgt, id, LLAMA_STATE_SEQ_FLAGS_NONE);
const size_t cur_size_dft = ctx_dft ? llama_state_seq_get_size_ext(ctx_dft, id, LLAMA_STATE_SEQ_FLAGS_NONE) : 0;
@@ -374,6 +376,8 @@ struct server_slot {
return false;
}
SLT_TRC(*this, "prompt cache save took %.2f ms\n", (ggml_time_us() - t_start) / 1000.0);
return true;
}
+4
View File
@@ -208,6 +208,8 @@ bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tok
if (it_best != states.end()) {
SRV_TRC(" - found better prompt with f_keep = %.3f, f_sim = %.3f\n", f_keep_best, f_sim_best);
const int64_t t_start = ggml_time_us();
{
auto & data = it_best->data.main;
@@ -252,6 +254,8 @@ bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tok
prompt = std::move(it_best->prompt);
states.erase(it_best);
SRV_TRC("slot %d: prompt cache restore took %.2f ms\n", id_slot, (ggml_time_us() - t_start) / 1000.0);
}
return true;
+13 -1129
View File
File diff suppressed because it is too large Load Diff
+3 -3
View File
@@ -110,8 +110,8 @@
"dompurify": "3.4.17",
"eslint": "10.14.0",
"eslint-config-prettier": "10.1.8",
"eslint-plugin-perfectionist": "^5.10.1",
"eslint-plugin-simple-import-sort": "^14.0.0",
"eslint-plugin-perfectionist": "5.13.0",
"eslint-plugin-simple-import-sort": "14.0.0",
"eslint-plugin-storybook": "10.6.6",
"eslint-plugin-svelte": "3.24.1",
"fflate": "0.8.3",
@@ -120,9 +120,9 @@
"http-server": "14.1.1",
"@ggml-org/llama-brand": "0.1.0",
"mdsvex": "0.12.7",
"mermaid": "11.17.2",
"mode-watcher": "1.1.0",
"pdfjs-dist": "5.4.54",
"pele": "1.13.0",
"playwright": "1.56.1",
"prettier": "3.8.3",
"prettier-plugin-svelte": "4.1.0",
+10 -2
View File
@@ -10,7 +10,9 @@ const RESOLVED_ID = '\0' + VIRTUAL_ID;
/**
* Bundle the vendored nerdamer-prime source into a minified IIFE string,
* exposed as the `virtual:nerdamer` module. Flags mirror the upstream
* loaded through the `virtual:nerdamer` module. In the client build it is
* its own file, fetched on first use, so it stays out of the app bundle.
* Flags mirror the upstream
* build (esbuild --bundle --minify --format=iife --global-name=nerdamer),
* so only human readable source lives in the repo and minification is a
* build artifact. Vendored under src/lib/vendors/, upstream snapshot:
@@ -41,7 +43,13 @@ export function nerdamerPlugin(): Plugin {
bundled = result.outputFiles[0].text;
}
return `export default ${JSON.stringify(bundled)};`;
if (this.environment.mode === 'build' && this.environment.name === 'client') {
const ref = this.emitFile({ name: 'nerdamer.js', source: bundled, type: 'asset' });
return `export default () => fetch(import.meta.ROLLUP_FILE_URL_${ref}).then((res) => res.text());`;
}
return `export default async () => ${JSON.stringify(bundled)};`;
},
name: 'llamacpp:nerdamer',
resolveId(id) {
+16 -2
View File
@@ -356,8 +356,22 @@
}
}
.mermaidTooltip {
display: none !important;
/*
* Mermaid diagrams are drawn by Pele, which reads its colors from these. They sit on :root
* so the svg also inherits them inside the preview dialog's shadow root, and follow the theme.
*/
:root {
--pele-bg: var(--code-background);
--pele-fg: var(--code-foreground);
--pele-muted: var(--muted-foreground);
--pele-line: var(--muted-foreground);
--pele-surface: var(--background);
--pele-surface-alt: var(--muted);
--pele-border: var(--border);
--pele-accent: var(--primary);
--pele-critical: var(--destructive);
--pele-error: var(--destructive);
--pele-radius: calc(var(--radius) - 4px);
}
/*
@@ -625,7 +625,7 @@
if ((hasRenderedBlocks || hasUnstableBlock) && containerRef) {
setupCodeBlockActions();
setupImageErrorHandlers();
renderMermaidDiagrams(containerRef, mode.current === ColorMode.DARK);
renderMermaidDiagrams(containerRef);
renderSvgDiagrams(containerRef);
}
});
@@ -626,10 +626,24 @@ div.markdown-user-content :global(.table-wrapper) {
/* Hide mermaid code text until rendered - prevents flash */
.markdown-content :global(pre.mermaid:not([data-mermaid-rendered])),
.markdown-content :global(pre.mermaid[data-mermaid-rendered]:not(:has(svg))) {
.markdown-content :global(pre.mermaid[data-mermaid-rendered]:not(:has(svg, .mermaid-error))) {
display: none;
}
/* A diagram Pele can't draw shows why, with the line it stopped at */
.markdown-content :global(.mermaid-error) {
max-width: 100%;
padding: 1rem;
color: var(--destructive);
font-family: var(--font-mono);
font-size: 0.8125rem;
/* the caret line under the source must line up character by character */
font-variant-ligatures: none;
text-align: left;
white-space: pre-wrap;
cursor: default;
}
/* Hide svg source until rendered - prevents flash. A rendered-but-unsanitized
block (oversized source) keeps its raw text visible as a safe fallback. */
.markdown-content :global(pre.svg-block:not([data-svg-rendered])) {
@@ -7,9 +7,15 @@ import {
DIAGRAM_VIEW_RENDERED,
DIAGRAM_VIEW_SOURCE,
MERMAID_BLOCK_CLASS,
MERMAID_ERROR_CLASS,
MERMAID_ID_ATTR,
MERMAID_RENDERED_ATTR,
MERMAID_SCROLL_CONTAINER_CLASS,
MERMAID_SVG_PADDING_X_REM,
MERMAID_SYNTAX_ATTR,
MERMAID_WRAPPER_CLASS,
PELE_SVG_CLASS,
PELE_THEME_VARIABLES,
SVG,
TOGGLE_SOURCE_BTN_CLASS
} from '#lib/constants/index.ts';
@@ -19,11 +25,30 @@ import { mountSvgShadow } from '#lib/utils/svg-shadow.ts';
export type DiagramAction = { copy: string } | { previewSvg: string };
// The width a diagram can take in its block: the scroll container's content box, less the
// svg's own horizontal padding. Undefined when the block isn't laid out (detached, hidden).
function diagramWidth(node: HTMLElement): number | undefined {
const scroll = node.closest<HTMLElement>(`.${MERMAID_SCROLL_CONTAINER_CLASS}`);
if (!scroll?.clientWidth) return undefined;
const box = getComputedStyle(scroll);
const rem = parseFloat(getComputedStyle(document.documentElement).fontSize);
const width =
scroll.clientWidth -
parseFloat(box.paddingLeft) -
parseFloat(box.paddingRight) -
MERMAID_SVG_PADDING_X_REM * rem;
return width > 0 ? width : undefined;
}
/**
* Renders the mermaid diagrams in `container` that haven't been rendered yet.
* Marks nodes immediately to prevent duplicate renders during streaming.
* Renders the mermaid diagrams in `container` that haven't been rendered yet, with Pele.
* Marks nodes immediately to prevent duplicate renders during streaming. Colors come
* from the `--pele-*` variables in app.css, so a diagram follows the theme without a redraw.
*/
export async function renderMermaidDiagrams(container: HTMLElement, isDark: boolean) {
export async function renderMermaidDiagrams(container: HTMLElement) {
const nodes = container.querySelectorAll<HTMLElement>(
`pre.${MERMAID_BLOCK_CLASS}:not([${MERMAID_RENDERED_ATTR}])`
);
@@ -34,32 +59,56 @@ export async function renderMermaidDiagrams(container: HTMLElement, isDark: bool
// This avoids needing a guard that would block node discovery.
nodes.forEach((node) => node.setAttribute(MERMAID_RENDERED_ATTR, BooleanString.TRUE));
// lazy load the mermaid dependecy only when needed to reduce bundle size.
const { default: mermaid } = await import('mermaid');
const { render } = await import('pele');
mermaid.initialize({
flowchart: {
htmlLabels: true,
useMaxWidth: false
},
gantt: {
useMaxWidth: false
},
securityLevel: 'strict',
sequence: {
useMaxWidth: false
},
startOnLoad: false,
theme: isDark ? 'dark' : 'default'
});
// Pele measures labels with the font it is given, so wait until the page font is in
await document.fonts?.ready;
try {
await mermaid.run({ nodes: Array.from(nodes) as unknown as NodeListOf<HTMLElement> });
} catch (error) {
console.error('Failed to render mermaid diagram:', error);
for (const node of nodes) {
const source = node.getAttribute(MERMAID_SYNTAX_ATTR) ?? node.textContent ?? '';
const id = node.getAttribute(MERMAID_ID_ATTR);
try {
const { svg } = render(source, {
fontFamily: getComputedStyle(node).fontFamily,
idPrefix: id ? id.replace(/[^\w-]/g, '_') : undefined,
// narrow blocks (phones) may turn a left-to-right diagram top-to-bottom
maxWidth: diagramWidth(node)
});
node.innerHTML = svg;
} catch (error) {
// the message quotes the diagram source, so it goes in as text
const message = document.createElement('div');
message.className = MERMAID_ERROR_CLASS;
message.textContent = error instanceof Error ? error.message : String(error);
node.replaceChildren(message);
}
}
}
/**
* A Pele diagram takes its colors from the page's `--pele-*` variables. Writes their current
* values into the svg, so a downloaded file keeps the colors it had on screen. Any other svg
* comes back unchanged.
*/
export function withThemeColors(svgHtml: string): string {
const svg = new DOMParser().parseFromString(svgHtml, 'image/svg+xml').documentElement;
if (!(svg instanceof SVGElement) || !svg.classList.contains(PELE_SVG_CLASS)) return svgHtml;
const theme = getComputedStyle(document.documentElement);
for (const name of PELE_THEME_VARIABLES) {
const value = theme.getPropertyValue(name).trim();
if (value) svg.style.setProperty(name, value);
}
return new XMLSerializer().serializeToString(svg);
}
/**
* Renders the svg diagrams in `container` that haven't been rendered yet.
* Sanitizes the source before injecting and marks each node so it renders once.
@@ -13,6 +13,7 @@ import { rehypeEnhanceMermaidBlocks } from './plugins/rehype/enhance-mermaid-blo
import { rehypeEnhanceSvgBlocks } from './plugins/rehype/enhance-svg-blocks';
import { rehypeEnhanceTables } from './plugins/rehype/enhance-tables';
import { rehypeFileBadge } from './plugins/rehype/file-badge';
import { rehypeLazyKatex } from './plugins/rehype/lazy-katex';
import { rehypeMermaidPre } from './plugins/rehype/mermaid-pre';
import { rehypeRtlSupport } from './plugins/rehype/rehype-rtl-support';
import { rehypeResolveAttachmentImages } from './plugins/rehype/resolve-attachment-images';
@@ -25,7 +26,6 @@ import type { Root as HastRoot } from 'hast';
import { all as lowlightAll } from 'lowlight';
import type { Root as MdastRoot } from 'mdast';
import rehypeHighlight from 'rehype-highlight';
import rehypeKatex from 'rehype-katex';
import rehypeStringify from 'rehype-stringify';
import { remark } from 'remark';
import remarkBreaks from 'remark-breaks';
@@ -71,7 +71,7 @@ function buildPipeline({
proc = proc.use(remarkRehype, allowHtml ? { allowDangerousHtml: true } : undefined);
if (!disableMath) {
proc = proc.use(rehypeKatex); // Render math using KaTeX
proc = proc.use(rehypeLazyKatex); // Render math using KaTeX, loaded on the first message with math
}
const pipeline = proc
@@ -0,0 +1,56 @@
/**
* Rehype plugin that renders math with rehype-katex, loading KaTeX only when a message has math.
*
* remark-math marks math as elements with a `language-math`, `math-inline` or `math-display`
* class; a tree without one is left alone, so a chat without math never downloads KaTeX.
*/
import type { Element, Root } from 'hast';
import type { Plugin, Transformer } from 'unified';
import { visit } from 'unist-util-visit';
const MATH_CLASSES = ['language-math', 'math-inline', 'math-display'];
let katexLoading: Promise<Transformer<Root>> | undefined;
let katexTransformer: Transformer<Root> | undefined;
function hasMath(tree: Root): boolean {
let found = false;
visit(tree, 'element', (node: Element) => {
const className = node.properties?.className;
if (Array.isArray(className) && className.some((c) => MATH_CLASSES.includes(String(c)))) {
found = true;
return false;
}
});
return found;
}
// synchronous when there is no math or KaTeX has loaded, so only the first message with math
// waits for it, and every other one renders in the same pass as before
export const rehypeLazyKatex: Plugin<[], Root> = () => {
return (tree, file) => {
if (!hasMath(tree)) return;
// rehype-katex changes the tree in place
if (katexTransformer) {
katexTransformer(tree, file, () => {});
return;
}
katexLoading ??= import('rehype-katex').then(({ default: rehypeKatex }) => {
katexTransformer = rehypeKatex() as Transformer<Root>;
return katexTransformer;
});
return katexLoading.then((transform) => {
transform(tree, file, () => {});
});
};
};
@@ -12,14 +12,12 @@ import {
import { getMarkdownProcessor } from '#lib/components/app/content/MarkdownContent/markdown-processor.ts';
import { getCodeInfoFromTarget } from '#lib/components/app/content/MarkdownContent/markdown-utils.ts';
import { MERMAID_BLOCK_CLASS } from '#lib/constants/index.ts';
import { ColorMode } from '#lib/enums/index.ts';
import {
copyCodeToClipboard,
copyToClipboard,
preprocessLaTeX,
splitGluedClosingCodeFences
} from '#lib/utils/index.ts';
import { mode } from 'mode-watcher';
export interface MarkdownBlockWidgetOptions {
onPreviewCode?: (code: string, language: string) => void;
@@ -311,9 +309,7 @@ export class MarkdownBlockWidget extends WidgetType {
// mermaid draws asynchronously and changes the block's height when done
if (dom.querySelector(`pre.${MERMAID_BLOCK_CLASS}`)) {
void renderMermaidDiagrams(dom, mode.current === ColorMode.DARK).then(() =>
this.measure(dom, view, key)
);
void renderMermaidDiagrams(dom).then(() => this.measure(dom, view, key));
}
}
}
@@ -3,6 +3,7 @@
import RotateCcwIcon from '@lucide/svelte/icons/rotate-ccw';
import ZoomInIcon from '@lucide/svelte/icons/zoom-in';
import ZoomOutIcon from '@lucide/svelte/icons/zoom-out';
import { withThemeColors } from '#lib/components/app/content/MarkdownContent/markdown-diagrams.ts';
import { ICON_CLASS_DEFAULT } from '#lib/constants/index.ts';
interface Props {
@@ -18,7 +19,7 @@
function downloadSvg() {
if (!svgHtml) return;
const blob = new Blob([svgHtml], { type: 'image/svg+xml' });
const blob = new Blob([withThemeColors(svgHtml)], { type: 'image/svg+xml' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
@@ -1,4 +1,9 @@
import { Compass, Eye, EyeOff, Heart, HeartOff, Trash2 } from '@lucide/svelte';
import Compass from '@lucide/svelte/icons/compass';
import Eye from '@lucide/svelte/icons/eye';
import EyeOff from '@lucide/svelte/icons/eye-off';
import Heart from '@lucide/svelte/icons/heart';
import HeartOff from '@lucide/svelte/icons/heart-off';
import Trash2 from '@lucide/svelte/icons/trash-2';
import { MODEL_DOWNLOAD_ICONS } from '#lib/constants/index.ts';
import { ModelRowDownloadState } from '#lib/enums/index.ts';
import { modelsStore, uiStore } from '#lib/stores/index.ts';
@@ -4,25 +4,23 @@
*/
import { MODALITY_FLAG_KEYS, MODALITY_ORDER } from '@ggml-org/llama-ui/models';
import {
CircleAlert as LoadFailedIcon,
Compass as DiscoverIcon,
Eject as UnloadIcon,
File as FileIcon,
FileText as FileTextIcon,
FolderOpen as CwdCommandIcon,
FoldVertical as CompactCommandIcon,
Image as ImageIcon,
Loader2 as LoadingIcon,
Mic as AudioIcon,
Package as ModelIcon,
Pause as PauseDownloadIcon,
Play as ResumeDownloadIcon,
Power as LoadIcon,
RotateCw as RetryIcon,
Sparkles as PromptCommandIcon,
Video as VideoIcon
} from '@lucide/svelte';
import LoadFailedIcon from '@lucide/svelte/icons/circle-alert';
import DiscoverIcon from '@lucide/svelte/icons/compass';
import UnloadIcon from '@lucide/svelte/icons/eject';
import FileIcon from '@lucide/svelte/icons/file';
import FileTextIcon from '@lucide/svelte/icons/file-text';
import CompactCommandIcon from '@lucide/svelte/icons/fold-vertical';
import CwdCommandIcon from '@lucide/svelte/icons/folder-open';
import ImageIcon from '@lucide/svelte/icons/image';
import LoadingIcon from '@lucide/svelte/icons/loader-2';
import AudioIcon from '@lucide/svelte/icons/mic';
import ModelIcon from '@lucide/svelte/icons/package';
import PauseDownloadIcon from '@lucide/svelte/icons/pause';
import ResumeDownloadIcon from '@lucide/svelte/icons/play';
import LoadIcon from '@lucide/svelte/icons/power';
import RetryIcon from '@lucide/svelte/icons/rotate-cw';
import PromptCommandIcon from '@lucide/svelte/icons/sparkles';
import VideoIcon from '@lucide/svelte/icons/video';
import { ChatFormCommandAction, FileTypeCategory, ModelCapability } from '#lib/enums/index.ts';
import type { ModelCapabilities } from '#lib/types/models.js';
import type { Component } from 'svelte';
@@ -1,4 +1,6 @@
import { Globe, Radio, Zap } from '@lucide/svelte';
import Globe from '@lucide/svelte/icons/globe';
import Radio from '@lucide/svelte/icons/radio';
import Zap from '@lucide/svelte/icons/zap';
import { MimeTypeImage } from '#lib/enums/files.enums.ts';
import { MCPTransportType } from '#lib/enums/index.ts';
import type { ClientCapabilities, Implementation } from '#lib/types/index.ts';
@@ -1,6 +1,25 @@
export const MERMAID_WRAPPER_CLASS = 'mermaid-block-wrapper';
export const MERMAID_SCROLL_CONTAINER_CLASS = 'mermaid-scroll-container';
export const MERMAID_BLOCK_CLASS = 'mermaid';
export const MERMAID_ERROR_CLASS = 'mermaid-error';
// left plus right padding of a diagram's svg, see `pre.mermaid svg` in markdown-content.css
export const MERMAID_SVG_PADDING_X_REM = 2;
// class Pele puts on the root of every diagram it draws
export const PELE_SVG_CLASS = 'pele';
// the Pele colors app.css maps to the theme, written into a downloaded diagram
export const PELE_THEME_VARIABLES = [
'--pele-bg',
'--pele-fg',
'--pele-muted',
'--pele-line',
'--pele-surface',
'--pele-surface-alt',
'--pele-border',
'--pele-accent',
'--pele-critical',
'--pele-radius'
] as const;
export const MERMAID_LANGUAGE = 'mermaid';
@@ -5,19 +5,17 @@ import { DEFAULT_MCP_CONFIG } from './mcp.constants';
import { SETTINGS_KEYS } from './settings-keys.constants';
import { TITLE_GENERATION } from './title-generation.constants';
import { FILE_GLOB_SEARCH_PICKERS } from './working-directory.constants';
import {
Code,
Database,
File,
Funnel,
GlassWater,
ListRestart,
Monitor,
Moon,
PencilRuler,
SlidersVertical,
Sun
} from '@lucide/svelte';
import Code from '@lucide/svelte/icons/code';
import Database from '@lucide/svelte/icons/database';
import File from '@lucide/svelte/icons/file';
import Funnel from '@lucide/svelte/icons/funnel';
import GlassWater from '@lucide/svelte/icons/glass-water';
import ListRestart from '@lucide/svelte/icons/list-restart';
import Monitor from '@lucide/svelte/icons/monitor';
import Moon from '@lucide/svelte/icons/moon';
import PencilRuler from '@lucide/svelte/icons/pencil-ruler';
import SlidersVertical from '@lucide/svelte/icons/sliders-vertical';
import Sun from '@lucide/svelte/icons/sun';
import { SurfaceTheme, SyncableParameterType } from '#lib/enums/index.ts';
import { SettingsFieldType } from '#lib/enums/settings.enums.ts';
import { ColorMode } from '#lib/enums/ui.enums.ts';
@@ -7,19 +7,17 @@
// (see ChatMessageToolCallBlockGetDatetime and
// ChatMessageToolCallBlockSearchResults for prior art).
import {
Braces,
Clock,
Eye,
FilePen,
FilePlus,
FileSearch,
FileText,
Info,
MessagesSquare,
SearchCode,
Terminal
} from '@lucide/svelte';
import Braces from '@lucide/svelte/icons/braces';
import Clock from '@lucide/svelte/icons/clock';
import Eye from '@lucide/svelte/icons/eye';
import FilePen from '@lucide/svelte/icons/file-pen';
import FilePlus from '@lucide/svelte/icons/file-plus';
import FileSearch from '@lucide/svelte/icons/file-search';
import FileText from '@lucide/svelte/icons/file-text';
import Info from '@lucide/svelte/icons/info';
import MessagesSquare from '@lucide/svelte/icons/messages-square';
import SearchCode from '@lucide/svelte/icons/search-code';
import Terminal from '@lucide/svelte/icons/terminal';
import { BuiltInTool, ToolSource } from '#lib/enums/index.ts';
import type { ToolUiEntry } from '#lib/types/index.ts';
@@ -6,7 +6,9 @@ import {
SHORTCUT_SETTINGS
} from './keyboard-shortcuts.constants';
import { MCP_SERVERS_ROUTE_ID, MODELS_ROUTE_ID, SETTINGS_ROUTE_ID } from './routes.constants';
import { Search, Settings, SquarePen } from '@lucide/svelte';
import Search from '@lucide/svelte/icons/search';
import Settings from '@lucide/svelte/icons/settings';
import SquarePen from '@lucide/svelte/icons/square-pen';
import McpLogo from '#lib/components/app/mcp/McpLogo.svelte';
import { SidebarAction, ToolSource } from '#lib/enums/index.ts';
import type { DesktopIconStripItem } from '#lib/types/index.ts';
@@ -33,9 +33,9 @@ async function getHarness(): Promise<string> {
if (!harnessCache[key]) {
if (enabled) {
const { default: nerdamerJs } = await import('virtual:nerdamer');
const { default: loadNerdamer } = await import('virtual:nerdamer');
harnessCache[key] = buildSandboxHarness(nerdamerJs);
harnessCache[key] = buildSandboxHarness(await loadNerdamer());
} else {
harnessCache[key] = buildSandboxHarness('');
}
+8 -10
View File
@@ -1,14 +1,12 @@
import { extractRootDomain } from './url';
import {
AlertTriangle,
Code,
Database,
File,
FileText,
Image,
Info,
XCircle
} from '@lucide/svelte';
import AlertTriangle from '@lucide/svelte/icons/alert-triangle';
import Code from '@lucide/svelte/icons/code';
import Database from '@lucide/svelte/icons/database';
import File from '@lucide/svelte/icons/file';
import FileText from '@lucide/svelte/icons/file-text';
import Image from '@lucide/svelte/icons/image';
import Info from '@lucide/svelte/icons/info';
import XCircle from '@lucide/svelte/icons/x-circle';
import {
CODE_FILE_EXTENSION_REGEX,
DEFAULT_RESOURCE_FILENAME,
+13 -13
View File
@@ -5,23 +5,23 @@
import { MimeTypeApplication, MimeTypeImage } from '#lib/enums/index.ts';
import { browser } from '$app/env';
import * as pdfjs from 'pdfjs-dist';
import pdfWorkerUrl from 'pdfjs-dist/build/pdf.worker.min.mjs?url';
type TextContent = {
items: Array<{ str: string }>;
};
if (browser) {
// Import worker as text and create blob URL for inline bundling
import('pdfjs-dist/build/pdf.worker.min.mjs?raw')
.then((workerModule) => {
const workerBlob = new Blob([workerModule.default], { type: 'application/javascript' });
let pdfjsModule: Promise<typeof import('pdfjs-dist')> | undefined;
pdfjs.GlobalWorkerOptions.workerSrc = URL.createObjectURL(workerBlob);
})
.catch(() => {
console.warn('Failed to load PDF.js worker, PDF processing may not work');
});
// pdf.js and its worker are files of their own, so the app loads them only when it opens a PDF
function loadPdfjs(): Promise<typeof import('pdfjs-dist')> {
pdfjsModule ??= import('pdfjs-dist').then((pdfjs) => {
pdfjs.GlobalWorkerOptions.workerSrc = pdfWorkerUrl;
return pdfjs;
});
return pdfjsModule;
}
/**
@@ -58,7 +58,7 @@ export async function convertPDFToText(file: File): Promise<string> {
}
try {
const buffer = await getFileAsBuffer(file);
const [pdfjs, buffer] = await Promise.all([loadPdfjs(), getFileAsBuffer(file)]);
const pdf = await pdfjs.getDocument({ data: buffer }).promise;
const numPages = pdf.numPages;
const textContentPromises: Promise<TextContent>[] = [];
@@ -96,7 +96,7 @@ export async function convertPDFToImage(file: File, scale: number = 1.5): Promis
}
try {
const buffer = await getFileAsBuffer(file);
const [pdfjs, buffer] = await Promise.all([loadPdfjs(), getFileAsBuffer(file)]);
const doc = await pdfjs.getDocument({ data: buffer }).promise;
const pages: Promise<string>[] = [];
+2 -2
View File
@@ -1,4 +1,4 @@
declare module 'virtual:nerdamer' {
const code: string;
export default code;
const loadNerdamer: () => Promise<string>;
export default loadNerdamer;
}
@@ -0,0 +1,48 @@
// pdf.js loads on first use, with its worker as a file of its own: extracting a PDF's text
// and drawing its pages go through that load.
import { convertPDFToImage, convertPDFToText } from '#lib/utils/pdf-processing.ts';
import { describe, expect, it } from 'vitest';
/** A one-page PDF that shows `text`, with a correct cross-reference table. */
function pdfWithText(text: string): File {
const stream = `BT /F1 24 Tf 72 720 Td (${text}) Tj ET`;
const objects = [
'<< /Type /Catalog /Pages 2 0 R >>',
'<< /Type /Pages /Kids [3 0 R] /Count 1 >>',
'<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>',
`<< /Length ${stream.length} >>\nstream\n${stream}\nendstream`,
'<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>'
];
let body = '%PDF-1.4\n';
const offsets: number[] = [];
objects.forEach((object, i) => {
offsets.push(body.length);
body += `${i + 1} 0 obj\n${object}\nendobj\n`;
});
const xref = body.length;
const entries = offsets.map((offset) => `${String(offset).padStart(10, '0')} 00000 n \n`);
body += `xref\n0 ${objects.length + 1}\n0000000000 65535 f \n${entries.join('')}`;
body += `trailer\n<< /Size ${objects.length + 1} /Root 1 0 R >>\nstartxref\n${xref}\n%%EOF\n`;
return new File([body], 'hello.pdf', { type: 'application/pdf' });
}
describe('pdf-processing', () => {
it('extracts the text of a PDF', async () => {
const text = await convertPDFToText(pdfWithText('Hello from a PDF'));
expect(text).toContain('Hello from a PDF');
});
it('draws each page of a PDF as a PNG', async () => {
const pages = await convertPDFToImage(pdfWithText('Page one'));
expect(pages).toHaveLength(1);
expect(pages[0]).toMatch(/^data:image\/png;base64,/);
});
});
+10 -8
View File
@@ -40,9 +40,10 @@ test.describe('PWA Service Worker', () => {
const swResponse = await page.request.get(swActive!);
const swContent = await swResponse.text();
// Precache contains SvelteKit content-hashed bundle paths
expect(swContent).toMatch(/["'`]_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js["'`]/);
expect(swContent).toMatch(/["'`]_app\/immutable\/assets\/bundle\.[a-zA-Z0-9_-]+\.css["'`]/);
// Precache contains SvelteKit's content-hashed entry points and CSS (a split build)
expect(swContent).toMatch(/["'`]_app\/immutable\/entry\/start\.[a-zA-Z0-9_-]+\.js["'`]/);
expect(swContent).toMatch(/["'`]_app\/immutable\/entry\/app\.[a-zA-Z0-9_-]+\.js["'`]/);
expect(swContent).toMatch(/["'`]_app\/immutable\/assets\/[a-zA-Z0-9_.-]+\.css["'`]/);
expect(swContent).toMatch(/["'`]manifest\.webmanifest["'`]/);
expect(swContent).toMatch(/["'`]_app\/version\.json["'`]/);
});
@@ -120,16 +121,17 @@ test.describe('PWA Service Worker', () => {
expect(manifest.icons.length).toBeGreaterThan(0);
});
test('index.html contains content-hashed bundle references', async ({ page }) => {
test('index.html contains content-hashed entry references', async ({ page }) => {
const response = await page.request.get('/');
expect(response.ok()).toBeTruthy();
const html = await response.text();
// SvelteKit outputs content-hashed bundle names in _app/immutable/
expect(html).toMatch(/href="(\.\/|\/)_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js"/);
expect(html).toMatch(/href="(\.\/|\/)_app\/immutable\/assets\/bundle\.[a-zA-Z0-9_-]+\.css"/);
expect(html).toMatch(/import\("(\.\/|\/)_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js"\)/);
// a split build: SvelteKit's content-hashed entry points in _app/immutable/entry/
expect(html).toMatch(/href="(\.\/|\/)_app\/immutable\/entry\/start\.[a-zA-Z0-9_-]+\.js"/);
expect(html).toMatch(/href="(\.\/|\/)_app\/immutable\/assets\/[a-zA-Z0-9_.-]+\.css"/);
expect(html).toMatch(/import\("(\.\/|\/)_app\/immutable\/entry\/start\.[a-zA-Z0-9_-]+\.js"\)/);
expect(html).toMatch(/import\("(\.\/|\/)_app\/immutable\/entry\/app\.[a-zA-Z0-9_-]+\.js"\)/);
});
});
@@ -1,7 +1,9 @@
<script lang="ts" module>
import { AI_TUTORIAL_MD } from '../fixtures/ai-tutorial.js';
import { MERMAID_DIAGRAMS_MD } from '../fixtures/mermaid-diagrams.js';
import { defineMeta } from '@storybook/addon-svelte-csf';
import { MarkdownContent } from '#lib/components/app/index.ts';
import { expect, waitFor } from 'storybook/test';
const { Story } = defineMeta({
component: MarkdownContent,
@@ -16,3 +18,23 @@
args={{ class: 'max-w-[56rem] w-[calc(100vw-2rem)]', content: AI_TUTORIAL_MD }}
name="Default"
/>
<Story
args={{ class: 'max-w-[56rem] w-[calc(100vw-2rem)]', content: MERMAID_DIAGRAMS_MD }}
name="Diagrams"
play={async ({ canvasElement }) => {
// MarkdownContent renders in batches, so the broken last block can trail the others
await waitFor(
() => {
expect(canvasElement.querySelectorAll('pre.mermaid svg.pele')).toHaveLength(4);
expect(canvasElement.querySelector('pre.mermaid .mermaid-error')).not.toBeNull();
},
{ timeout: 5000 }
);
const error = canvasElement.querySelector('pre.mermaid .mermaid-error');
await expect(error?.textContent).toContain('line 2');
await expect(error?.closest('pre')).toBeVisible();
}}
/>
@@ -0,0 +1,54 @@
// Mermaid blocks of several diagram types, and one that can't be drawn
const fence = '```';
export const MERMAID_DIAGRAMS_MD = String.raw`
How a request reaches the model:
${fence}mermaid
flowchart LR
U[Browser] -->|POST /v1/chat/completions| S(llama-server)
S --> Q{Slot free?}
Q -->|yes| D[Decode]
Q -->|no| W[Queue]
W --> D
D -->|SSE tokens| U
${fence}
The same exchange over time:
${fence}mermaid
sequenceDiagram
participant C as Llama Chat
participant S as llama-server
C->>S: POST /v1/chat/completions
S-->>C: data: {"choices":[...]}
S-->>C: data: [DONE]
Note over C,S: the stream ends with DONE
${fence}
A slot's life:
${fence}mermaid
stateDiagram-v2
[*] --> Idle
Idle --> Processing: prompt
Processing --> Generating: prompt evaluated
Generating --> Idle: stop or limit
${fence}
Where the time goes:
${fence}mermaid
pie title Time per request
"Prompt processing" : 35
"Token generation" : 60
"Sampling" : 5
${fence}
A diagram with a mistake in it:
${fence}mermaid
flowchart LR
A -->
${fence}
`;
@@ -0,0 +1,32 @@
import { rehypeLazyKatex } from '#lib/components/app/content/MarkdownContent/plugins/rehype/lazy-katex.ts';
import type { Root } from 'hast';
import rehypeStringify from 'rehype-stringify';
import { remark } from 'remark';
import remarkMath from 'remark-math';
import remarkRehype from 'remark-rehype';
import { describe, expect, it } from 'vitest';
const processor = remark()
.use(remarkMath)
.use(remarkRehype)
.use(rehypeLazyKatex)
.use(rehypeStringify);
async function render(markdown: string): Promise<string> {
return processor.stringify((await processor.run(processor.parse(markdown))) as Root);
}
describe('rehypeLazyKatex', () => {
it('renders inline and display math with KaTeX', async () => {
const html = await render('Area $\\pi r^2$\n\n$$\nE = mc^2\n$$');
expect(html.match(/class="katex"/g)).toHaveLength(2);
expect(html).toContain('katex-display');
});
it('leaves a message without math as it is, in the same pass', () => {
const tree = processor.runSync(processor.parse('Just *text* and `code`')) as Root;
expect(processor.stringify(tree)).toBe('<p>Just <em>text</em> and <code>code</code></p>');
});
});
+32 -26
View File
@@ -38,24 +38,23 @@ describe('PWA Build Output', () => {
).toBeTruthy();
});
it('SvelteKit bundle.js exists in _app/immutable/', () => {
// SvelteKit generates hashed bundle names in _app/immutable/
const appDir = resolve(DIST_DIR, '_app', 'immutable');
it('SvelteKit entry start.js and app.js exist in _app/immutable/entry/', () => {
// a split build: hashed entry points in _app/immutable/entry/, the rest in chunks/ and nodes/
const entryDir = resolve(DIST_DIR, '_app', 'immutable', 'entry');
expect(existsSync(appDir), '_app/immutable/ not found').toBeTruthy();
const files = readdirSync(appDir).filter((f) => f.startsWith('bundle.') && f.endsWith('.js'));
expect(existsSync(entryDir), '_app/immutable/entry/ not found').toBeTruthy();
const files = readdirSync(entryDir);
expect(files.length).toBeGreaterThan(0);
expect(files.some((f) => /^start\.[a-zA-Z0-9_-]+\.js$/.test(f))).toBe(true);
expect(files.some((f) => /^app\.[a-zA-Z0-9_-]+\.js$/.test(f))).toBe(true);
});
it('SvelteKit bundle.css exists in _app/immutable/assets/', () => {
// SvelteKit generates hashed CSS bundles in _app/immutable/assets/
it('SvelteKit CSS exists in _app/immutable/assets/', () => {
// SvelteKit generates hashed CSS files in _app/immutable/assets/
const cssDir = resolve(DIST_DIR, '_app', 'immutable', 'assets');
expect(existsSync(cssDir), '_app/immutable/assets/ not found').toBeTruthy();
const files = readdirSync(cssDir).filter(
(f) => f.startsWith('bundle.') && f.endsWith('.css')
);
const files = readdirSync(cssDir).filter((f) => f.endsWith('.css'));
expect(files.length).toBeGreaterThan(0);
});
@@ -81,12 +80,15 @@ describe('PWA Build Output', () => {
});
describe('Service worker content', () => {
it('precache contains SvelteKit bundle.js with content hash', () => {
expect(swContent).toMatch(/["'`]_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js["'`]/);
});
it('precache contains every file in _app/immutable/, so lazily loaded chunks work offline', () => {
const immutableDir = resolve(DIST_DIR, '_app', 'immutable');
const files = readdirSync(immutableDir, { recursive: true, withFileTypes: true })
.filter((entry) => entry.isFile())
.map((entry) => resolve(entry.parentPath, entry.name).slice(resolve(DIST_DIR).length + 1));
it('precache contains SvelteKit bundle.css with content hash', () => {
expect(swContent).toMatch(/["'`]_app\/immutable\/assets\/bundle\.[a-zA-Z0-9_-]+\.css["'`]/);
expect(files.length).toBeGreaterThan(0);
for (const file of files) expect(swContent).toContain(file);
});
it('precache contains the app shell, _app/version.json and manifest.webmanifest', () => {
@@ -119,23 +121,27 @@ describe('PWA Build Output', () => {
});
describe('index.html content', () => {
it('has modulepreload link for SvelteKit bundle with content hash', () => {
expect(indexContent).toBeTruthy();
// SvelteKit generates hashed bundle names in _app/immutable/
expect(indexContent).toMatch(/href="(\.\/|\/)_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js"/);
});
it('has stylesheet link for SvelteKit bundle.css with content hash', () => {
it('has modulepreload link for the SvelteKit entry with content hash', () => {
expect(indexContent).toBeTruthy();
expect(indexContent).toMatch(
/href="(\.\/|\/)_app\/immutable\/assets\/bundle\.[a-zA-Z0-9_-]+\.css"/
/href="(\.\/|\/)_app\/immutable\/entry\/start\.[a-zA-Z0-9_-]+\.js"/
);
});
it('has dynamic import for SvelteKit bundle with content hash', () => {
it('has stylesheet link for SvelteKit CSS with content hash', () => {
expect(indexContent).toBeTruthy();
expect(indexContent).toMatch(
/import\("(\.\/|\/)_app\/immutable\/bundle\.[a-zA-Z0-9_-]+\.js"\)/
/href="(\.\/|\/)_app\/immutable\/assets\/[a-zA-Z0-9_.-]+\.[a-zA-Z0-9_-]+\.css"/
);
});
it('has dynamic imports for the SvelteKit entry with content hash', () => {
expect(indexContent).toBeTruthy();
expect(indexContent).toMatch(
/import\("(\.\/|\/)_app\/immutable\/entry\/start\.[a-zA-Z0-9_-]+\.js"\)/
);
expect(indexContent).toMatch(
/import\("(\.\/|\/)_app\/immutable\/entry\/app\.[a-zA-Z0-9_-]+\.js"\)/
);
});
+2
View File
@@ -28,6 +28,8 @@
"highlight.js/*": ["./node_modules/highlight.js/*"],
"@ggml-org/llama-brand": ["./node_modules/@ggml-org/llama-brand"],
"@ggml-org/llama-brand/*": ["./node_modules/@ggml-org/llama-brand/*"],
"lowlight": ["./node_modules/lowlight"],
"lowlight/*": ["./node_modules/lowlight/*"],
"mode-watcher": ["./node_modules/mode-watcher"],
"mode-watcher/*": ["./node_modules/mode-watcher/*"],
"svelte": ["./node_modules/svelte"],
+1 -1
View File
@@ -55,7 +55,7 @@ export default defineConfig(({ mode }) => {
strict: true
}),
extensions: ['.svelte', '.svx'],
output: { bundleStrategy: 'single' },
output: { bundleStrategy: 'split' },
paths: { relative: true },
// Consult https://svelte.dev/docs/kit/integrations
// for more information about preprocessors
+60 -23
View File
@@ -10,13 +10,14 @@
"license": "MIT",
"dependencies": {
"@ggml-org/llama-brand": "0.1.0",
"@internationalized/date": "^3.12.4",
"@internationalized/date": "3.12.4",
"@lucide/svelte": "1.46.0",
"bits-ui": "2.18.1",
"clsx": "^2.1.1",
"highlight.js": "^11.11.1",
"tailwind-merge": "^3.7.0",
"tailwind-variants": "^3.3.1",
"clsx": "2.1.1",
"highlight.js": "11.11.1",
"lowlight": "3.3.0",
"tailwind-merge": "3.7.0",
"tailwind-variants": "3.3.1",
"vaul-svelte": "1.0.0-next.7"
},
"devDependencies": {
@@ -27,28 +28,28 @@
"@storybook/addon-svelte-csf": "5.1.6",
"@storybook/addon-vitest": "10.6.6",
"@storybook/sveltekit": "10.6.6",
"@sveltejs/adapter-auto": "^8.0.0",
"@sveltejs/kit": "^3.0.0",
"@sveltejs/package": "^2.5.8",
"@sveltejs/vite-plugin-svelte": "^7.2.0",
"@tailwindcss/vite": "^4.3.3",
"@types/node": "^24",
"@sveltejs/adapter-auto": "8.0.0",
"@sveltejs/kit": "3.2.0",
"@sveltejs/package": "2.5.8",
"@sveltejs/vite-plugin-svelte": "7.3.1",
"@tailwindcss/vite": "4.3.3",
"@types/node": "24.19.2",
"@vitest/browser-playwright": "5.0.5",
"@vitest/coverage-v8": "5.0.5",
"mdsvex": "0.12.7",
"mode-watcher": "^1.1.0",
"playwright": "^1.60.0",
"publint": "^0.3.21",
"mode-watcher": "1.1.0",
"playwright": "1.64.1",
"publint": "0.3.25",
"storybook": "10.6.6",
"svelte": "^5.57.1",
"svelte-check": "^4.6.0",
"tailwindcss": "^4.3.3",
"svelte": "5.57.3",
"svelte-check": "4.7.7",
"tailwindcss": "4.3.3",
"tw-animate-css": "1.4.0",
"typescript": "^6.0.3",
"typescript": "6.0.3",
"vite": "npm:@voidzero-dev/vite-plus-core@1.3.0",
"vite-plus": "1.3.0",
"vitest": "5.0.5",
"vitest-browser-svelte": "^2.1.1"
"vitest-browser-svelte": "2.2.1"
},
"peerDependencies": {
"mode-watcher": "^1.1.0",
@@ -3578,6 +3579,15 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/hast": {
"version": "3.0.5",
"resolved": "https://registry.npmjs.org/@types/hast/-/hast-3.0.5.tgz",
"integrity": "sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==",
"license": "MIT",
"dependencies": {
"@types/unist": "*"
}
},
"node_modules/@types/json-schema": {
"version": "7.0.15",
"resolved": "https://registry.npmjs.org/@types/json-schema/-/json-schema-7.0.15.tgz",
@@ -3616,7 +3626,6 @@
"version": "2.0.11",
"resolved": "https://registry.npmjs.org/@types/unist/-/unist-2.0.11.tgz",
"integrity": "sha512-CmBKiL6NNo/OqgmMn95Fk9Whlp2mtvIv+KNpQKN2F4SjvrEesubTRWGYSg+BnWZOnlCaSTU1sMpsBOzgbYhnsA==",
"dev": true,
"license": "MIT"
},
"node_modules/@valibot/to-json-schema": {
@@ -4707,6 +4716,19 @@
"dev": true,
"license": "MIT"
},
"node_modules/devlop": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/devlop/-/devlop-1.1.0.tgz",
"integrity": "sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==",
"license": "MIT",
"dependencies": {
"dequal": "^2.0.0"
},
"funding": {
"type": "github",
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/dom-accessibility-api": {
"version": "0.5.16",
"resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.5.16.tgz",
@@ -5329,6 +5351,21 @@
"dev": true,
"license": "MIT"
},
"node_modules/lowlight": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/lowlight/-/lowlight-3.3.0.tgz",
"integrity": "sha512-0JNhgFoPvP6U6lE/UdVsSq99tn6DhjjpAj5MxG49ewd2mOBVtwWYIT8ClyABhq198aXXODMU6Ox8DrGy/CpTZQ==",
"license": "MIT",
"dependencies": {
"@types/hast": "^3.0.0",
"devlop": "^1.0.0",
"highlight.js": "~11.11.0"
},
"funding": {
"type": "github",
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/lz-string": {
"version": "1.5.0",
"resolved": "https://registry.npmjs.org/lz-string/-/lz-string-1.5.0.tgz",
@@ -6149,9 +6186,9 @@
}
},
"node_modules/svelte": {
"version": "5.57.2",
"resolved": "https://registry.npmjs.org/svelte/-/svelte-5.57.2.tgz",
"integrity": "sha512-1YiPCAXT0Ct6g1vOMmSLa0Nsc3QRMvtDF32ihgglGVrugIuIK38RyDiFNXVDhtQ81TJjSO57NdMw6DyYLqSGDQ==",
"version": "5.57.3",
"resolved": "https://registry.npmjs.org/svelte/-/svelte-5.57.3.tgz",
"integrity": "sha512-5mAftM3SoDV9X0zdF386F9p2pv8iKOwi/KfBS1XnBAkVCKFzVOLHUoa8iGWAt4buoicLmyWVRpuuD5wplXOQsg==",
"dev": true,
"license": "MIT",
"dependencies": {
+20 -19
View File
@@ -86,28 +86,28 @@
"@storybook/addon-svelte-csf": "5.1.6",
"@storybook/addon-vitest": "10.6.6",
"@storybook/sveltekit": "10.6.6",
"@sveltejs/adapter-auto": "^8.0.0",
"@sveltejs/kit": "^3.0.0",
"@sveltejs/package": "^2.5.8",
"@sveltejs/vite-plugin-svelte": "^7.2.0",
"@tailwindcss/vite": "^4.3.3",
"@types/node": "^24",
"@sveltejs/adapter-auto": "8.0.0",
"@sveltejs/kit": "3.2.0",
"@sveltejs/package": "2.5.8",
"@sveltejs/vite-plugin-svelte": "7.3.1",
"@tailwindcss/vite": "4.3.3",
"@types/node": "24.19.2",
"@vitest/browser-playwright": "5.0.5",
"@vitest/coverage-v8": "5.0.5",
"mdsvex": "0.12.7",
"mode-watcher": "^1.1.0",
"playwright": "^1.60.0",
"publint": "^0.3.21",
"mode-watcher": "1.1.0",
"playwright": "1.64.1",
"publint": "0.3.25",
"storybook": "10.6.6",
"svelte": "^5.57.1",
"svelte-check": "^4.6.0",
"tailwindcss": "^4.3.3",
"svelte": "5.57.3",
"svelte-check": "4.7.7",
"tailwindcss": "4.3.3",
"tw-animate-css": "1.4.0",
"typescript": "^6.0.3",
"typescript": "6.0.3",
"vite": "npm:@voidzero-dev/vite-plus-core@1.3.0",
"vite-plus": "1.3.0",
"vitest": "5.0.5",
"vitest-browser-svelte": "^2.1.1"
"vitest-browser-svelte": "2.2.1"
},
"peerDependencies": {
"mode-watcher": "^1.1.0",
@@ -120,14 +120,15 @@
"vitest": "5.0.5"
},
"dependencies": {
"@internationalized/date": "^3.12.4",
"@internationalized/date": "3.12.4",
"@lucide/svelte": "1.46.0",
"bits-ui": "2.18.1",
"clsx": "^2.1.1",
"highlight.js": "^11.11.1",
"clsx": "2.1.1",
"highlight.js": "11.11.1",
"lowlight": "3.3.0",
"@ggml-org/llama-brand": "0.1.0",
"tailwind-merge": "^3.7.0",
"tailwind-variants": "^3.3.1",
"tailwind-merge": "3.7.0",
"tailwind-variants": "3.3.1",
"vaul-svelte": "1.0.0-next.7"
}
}
+5 -7
View File
@@ -2,13 +2,11 @@
// models-discover-download and model-compatibility).
import { ModelCapability, ModelModality } from './enums.js';
import type { ModelModalities } from './types.js';
import {
Eye as VisionIcon,
Lightbulb as ReasoningIcon,
Mic as AudioIcon,
Video as VideoIcon,
Wrench as ToolUseIcon
} from '@lucide/svelte';
import VisionIcon from '@lucide/svelte/icons/eye';
import ReasoningIcon from '@lucide/svelte/icons/lightbulb';
import AudioIcon from '@lucide/svelte/icons/mic';
import VideoIcon from '@lucide/svelte/icons/video';
import ToolUseIcon from '@lucide/svelte/icons/wrench';
import type { Component } from 'svelte';
/** Neutral model badge: params, quantization, tags. */
+5 -1
View File
@@ -1,7 +1,11 @@
// highlightCode and its helpers, from ui/apps/chat's utils/code.ts (only what SyntaxHighlightedCode uses)
import { CODE_BLOCK } from "../constants";
import hljs from "highlight.js";
import hljs from "highlight.js/lib/core";
import bash from "highlight.js/lib/languages/bash";
import { all as languages } from "lowlight";
// reuse lowlight's language modules: the full `highlight.js` entry bundles a second, CommonJS copy of every language
for (const [name, language] of Object.entries(languages)) hljs.registerLanguage(name, language);
// highlight.js's bash marks strings, variables and keywords but leaves a command's name and its flags plain, so a
// command (`curl -H … -d …`) reads as one colour. Mark them too: the name as a built-in, the flags as attributes.