Compare commits

..
11 Commits
Author SHA1 Message Date
Johannes Gäßler b94959a64d CUDA: implement support for duplicate expert IDs (#32505)
implement support for duplicate expert IDs
2026-11-19 23:21:24 +01:00
Imdadullah Mohammed 50518b64c6 model : hrm : do not compute slot count as uint32_t (#32601)
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.

Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.

load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
2026-11-19 23:21:03 +01:00
c406f4367e opencl: support dp4a for KQ mm in decomposed FA (#32577)
* opencl: support int8 KQ for DK = 512

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: allow query chunk up to 96 for int8 KQ

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

* opencl: extend `build_program_from_source` to allow non-fatal failure

Assisted-by: Codex

* opencl: add int8 KQ for decomposed FA with DK <= 256

Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Assisted-by: Codex

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-19 11:30:19 -08:00
anujj 7fce5439fe ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA) (#32387)
* ggml : add BF16 support to ROPE and SET_ROWS (CPU, CUDA)

* cuda : SET_ROWS from BF16 into quantized types and F8_E4M3

* cuda : fuse BF16 rope into a BF16 cache

* tests : trim the BF16 rope cases to one per code path
2026-11-19 23:43:55 +05:30
JamePeng 53ecea05a0 mtmd : fix YuE2 projector loading (#32606)
- Allow cls.output.weight to be absent from the mmproj. It is an optional
Qwen3 reranking head that the YuE2 NAR decoder never uses.

- Without this fix, mtmd_init_from_file() fails with
"tensor cls.output.weight is missing from the mmproj", preventing
audio generation from starting.
2026-11-19 19:11:11 +01:00
9a080f44b5 ggml-openvino: fix OpenVINO CI, minor fixes and optimizations (#32590)
* ggml-openvino: round the GPU KV length up to a power of two

Review notes (follow-ups, not blocking):
- convert_ggml_input_to_ov pads the KQ mask into a std::vector and then
  copies it into the ov::Tensor. Writing the padded rows straight into
  padded.data() avoids the second allocation and copy (up to 16 MB per
  call at a 16K context).
- The padding is skipped when the mask is not contiguous, and the KV
  inputs are then sliced to more rows than the mask has columns. Masks
  are always contiguous today; a GGML_ASSERT would make that explicit.
- compute_llm_params reads GGML_OPENVINO_STATEFUL_EXECUTION directly;
  is_stateful_enabled() is local to ggml-openvino.cpp.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Optimize model recompiles of prefill stage

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.

* ggml-openvino: Remove solution C and keep A as default

* ggml-openvino: match the output count before reusing a compiled NPU prefill

The static prefill model has a fixed inp_out_ids shape [1,1,1,output_len].
The new reuse check compared only the attention sizes, so llama-perplexity,
which alternates 1 and 512 outputs per ubatch, reused the model compiled
for 512 outputs on a 1-output ubatch and failed with "got [1,1,1,1]
expecting [1,1,1,512]". Store the output count of the cached prefill
model and require it to match.

* ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding

* ggml-openvino: remove leftovers from merging dev into the LFM2/DSpark fix

- compute_model_outputs exported OUTPUT-flagged nodes twice: dev's block
  ran before the new one and still exported result_norm, so the
  result_norm exclusion had no effect. Keep only the new block.
- compute_node_dynamic_dims checked is_activation_input twice in a row.
  Remove the second copy.
- process_view_input_new created the dynamic Split on split_dim. With
  stateful execution the input drops the leading size-1 axis, so use
  split_axis, which already accounts for the rank difference.

* ggml-openvino: hash the output flags into the model-cache fingerprint

The fingerprint hashed the ops, names and weights but not the outputs, so
a blob compiled with another output set was imported. With the result_norm
change, an NPU blob written before it gave wrong output, and enabling
llama_set_embeddings_layer_inp() reused a blob without the layer-input
outputs. Hash each node's output flag and bump the cache version to 8.

* ggml-openvino: support f16 SCALE and f16/bf16 TRI

test-backend-ops now runs SCALE on f16 and TRI on f16 and bf16. Both
translators built their constants as f32, and Multiply, Add and Select
need all inputs in one element type, so building the graph threw for
these cases.

SCALE now converts a non-f32 input to f32, applies the scale and bias,
and converts back once, as the CPU backend computes it. TRI creates its
zero constant in the input's element type. f32 graphs are unchanged.

* ggml-openvino: [Qwen3.6 A3B]fix static-shape ids view of the last MoE
 layer

The same-stride single-dim Slice in process_view_input_new was followed by a Reshape
whose target baked the static chunk size into the token axis. After the output-ids
GET_ROWS the last layer runs on n_outputs tokens (1 in the prefill graph), so the
Reshape failed shape validation for qwen35moe with FORCE_STATIC or on the NPU.
Skip the Reshape in static graphs when the slice already keeps the rank;
the dynamic graphs are unchanged.

* enable mamba based models on OV CPU-GPU

* revert changes in tests

* docs : add Falcon-H1 to the OpenVINO validated models

Falcon-H1 runs on CPU and GPU since Mamba SSM support was added. Add its
row and the known limitations: multi-sequence batches fail, GPU accuracy
is slightly lower than CPU, and the NPU fails with an element type
mismatch.

* ggml-openvino: simplify the SSM_SCAN build check and clean up

- Drop the SelectiveSSM configure check and its guards: OpenVINO 2026.4.1
  always provides the op. Drop the no-op LANGUAGE CXX source property.
- Drop the semicolon after OPENVINO_OP("SelectiveSSM"). GCC 15 reports it
  with -Wextra-semi, which fails builds with fatal warnings.
- Add ggml_openvino_is_static() for the repeated NPU or
  GGML_OPENVINO_FORCE_STATIC check.
- Share the f16 and f32 KQ mask padding code.
- Drop the redundant inline on in-class static functions.

---------

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
Co-authored-by: yzhou51 <yaowei.zhou@intel.com>
Co-authored-by: Ananth, Swathi <swathi.ananth@intel.com>
Co-authored-by: Beheshti <nazanin.beheshti@intel.com>
2026-11-19 07:22:26 -08:00
Bhargav Krish c7468107ef cuda: Scatter kernel for set_n (#32489)
* cuda: add scatter for batched tensor writes

* update test case to mirror get_n test case

* save restore timing logs

* refactor code into batch-copy.cu file
2026-11-19 20:14:47 +05:30
ac830a9cbb ggml : add backend scheduler tests (#27258)
* Adds initial scheduler stress tests

* Adds new sleep op to simplify scheduler testing

* Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321

* Adds experimental vulkan GGML_OP_SLEEP

* Improves structure and naming of previous tests. Adds new tests,
including one which detects the nkvo-bug correctly, and validates the
proposed hotfix.

* Cleanup: Removes CUDA specific comments and nvtx-markers

* Fix: forgot to add sleep specific test case

* Fix oversights identified by CI

* Drops default prints so that it behaves like other ctests. Adds -v flag
to toggle this behavior

* Fix: Early exit in CPU-only settings (cause for graviton CI fails)

* Improves stress_test_linked_list by removing overlap with
test_chain_all_backends and test_pair_user_inputs. The latter ones test
either activations or user inputs in isolation and just once, whereas
the stress test tests all combinations multiple rounds

* Exclude ACCEL devices from new tests due to incompatibility

* Fixes vulkan test failure in llvmpipe

* Adds asserts, improves comments and some checks

* Fixes nitpicks in white-space and variable naming

* Adds comments that certain vk drivers could time OP_SLEEP incorrectly

* Drops limitiation to CPU/GPU in test-sleep

* Aligns OP_SLEEP guards in cuda with vk guards (input shape and
contiguity)

* Implements feedback for OP_SLEEP to work in place (no copy)

* Apply non-code suggestions  from JohannesGaesslers code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Removes struct backend_supports; enables tests to return skip, instead
of only pass/fail

* Make clang-tidy happy

* Removes sched_backend_caps, improves create_test_scheduler according to
feedback (L149)

* Removes "name" field from struct sched_check in favor of
ggml_set_name/ggml_get_name

* Drops unnecessary shader_clock.comp test and guards

* Replaces sched_ prefixes with helper_ prefix

* Moves test suite to new RAII types for better memory management

* Adds sleep option to test_chain_all_backends

* Adds sleep option to first-run arm of the y-shaped test

* Fix switchcase order for legibility in stress_test_dag

* Simplfies stress_test_linked_list via double for-loop

* Adds pseudo-random input to stress_test_linked_list, and runs all
transitions both with and without CPU input (next to activation copy as
input)

* Fix RPC Proto patch due to GGML_OPS bump

* Adds ggml-cpp.h treatment to test-sleep.cpp

* Adds TODO pointers for subsequent additional tests.

* Apply suggestion to ggml.h

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Apply suggestion to tests/test-backend-sched.cpp

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Fixes node compatibility check to not take buffer locations into account.
This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test)

* Apply batched suggestions from gg code review

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Completes callsite change of gg-suggestion

* Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us
to avoid mac clock drift and spurious test failures.
This makes the test a bit nonsensical on CPU (same clock checks itself),
but the test still has merit on the other backends where
independent device clocks are used.

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-11-19 16:40:13 +02:00
Piotr Wilkin (ilintar) 8af68d486a llama: fixes for Inkling 2 attention mask (#32429)
* llama : give Inkling2 its own relative-position attention input

The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.

Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.

Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* inkling2 : make the expert weight scale and final logit softcap optional

Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : build kv_rows masks for the hybrid iswa memory

llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : make the relative-position input a hybrid iswa input

Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>

* llama : compose the hybrid iswa rel input, format inkling2

Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.

Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.

Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 14:36:30 +01:00
Alexey Kopytko f8caa5ea30 metal : fix the second argument of NSMakeRange (#32602) 2026-11-19 13:49:22 +02:00
SS1618 4db8b7638f webgpu: add POOL_2D support (#32250)
* Shader, pipeline, and launcher

* Update docs and add newline to wgsl

* Formatting fixes
2026-11-19 13:35:10 +02:00
70 changed files with 4120 additions and 829 deletions
+3
View File
@@ -195,6 +195,7 @@ The models below were tested with `llama-cli` and `Q4_K_M` files on an Intel® C
| [bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M](https://huggingface.co/bartowski/prism-ml_Bonsai-8B-unpacked-GGUF) | ✓ | ✓ | ✓ |
| [LiquidAI/LFM2.5-2.6B-Q4_K_M](https://huggingface.co/LiquidAI/LFM2.5-2.6B-GGUF) | ✓ | ✓ | ✗ |
| [bartowski/inclusionAI_Ling-mini-2.0-Q4_K_M](https://huggingface.co/bartowski/inclusionAI_Ling-mini-2.0-GGUF) | ✓ | ✓ | ✗ |
| [tiiuae/Falcon-H1-0.5B-Instruct-Q4_K_M](https://huggingface.co/tiiuae/Falcon-H1-0.5B-Instruct-GGUF) | ✓ | ✓ | ✗ |
| | | | |
| [Qwen/Qwen3-Embedding-0.6B-Q8_0](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF) | ✓ | ✓ | ✗ |
| [ggml-org/bge-m3-Q8_0](https://huggingface.co/ggml-org/bge-m3-Q8_0-GGUF) | ✓ | ✓ | ✗ |
@@ -990,6 +991,7 @@ build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.
- Multimodal features (audio/image/video) are a work in progress.
- Embedding models: reranking models are not validated.
- Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.
- Mamba-based models (e.g. Falcon-H1): `llama-perplexity` with several sequences per batch fails; set `-b` to the same value as `-c`. GPU accuracy is slightly lower than CPU.
**GPU-specific**
@@ -1012,6 +1014,7 @@ build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.
- `llama-perplexity`: requires `-b 512` or smaller.
- Flash attention must stay on, which is the default (`-fa auto`). With `-fa off`, the run fails on the NPU.
- `llama-embedding` fails on the NPU.
- Mamba-based models (e.g. Falcon-H1) fail on the NPU with `Failed to set tensor` (element type mismatch).
- Lunar Lake NPU: decode slows down sharply when the context is large. The context size at which this starts depends on the model.
- **Workaround:** Pass the smallest context that fits the use, e.g. `-c 512`, or `-c 256` for larger models.
- Panther Lake NPU: the load of a Qwen3.5 model can end with a segmentation fault in the NPU compiler of OpenVINO 2026.4.1.
+1 -1
View File
@@ -90,7 +90,7 @@ Legend:
| PAD | ❌ | 🟡 | ✅ | 🟡 | ❌ | ✅ | 🟡 | ✅ | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ |
| PAD_REFLECT_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ |
| POOL_1D | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
| POOL_2D | ❌ | 🟡 | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
| REGLU | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ❌ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| RELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
| REPEAT | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | 🟡 | ❌ | ❌ |
+128 -128
View File
@@ -1155,134 +1155,134 @@
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i32,ne=[1,8,1,3],nr23=[1,1],r=2,v=0","support","0","no","WebGPU"
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i64,ne=[1,8,1,3],nr23=[1,1],r=2,v=1","support","0","no","WebGPU"
"WebGPU: WebGPU","SET_ROWS","type_src=f16,type_dst=f16,type_idx=i32,ne=[1,8,1,3],nr23=[1,1],r=2,v=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=avg,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=1,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=1,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=1,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=1,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=0,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=0","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_2D","pool_type=max,type_input=f32,ne_input=[10,10,3,1],k0=3,k1=3,s0=2,s1=2,p0=1,p1=1","support","1","yes","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[10,3,2,1],k0=1,s0=1,p0=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[11,1,3,2],k0=1,s0=1,p0=0","support","0","no","WebGPU"
"WebGPU: WebGPU","POOL_1D","pool_type=avg,type_input=f32,ne_input=[128,2,1,3],k0=1,s0=1,p0=0","support","0","no","WebGPU"
Can't render this file because it is too large.
+2 -2
View File
@@ -8,10 +8,10 @@ extern "C" {
#define RPC_PROTO_MAJOR_VERSION 10
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_PATCH_VERSION 0
#define RPC_PROTO_PATCH_VERSION 1
#ifdef __cplusplus
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
#endif
#define GGML_RPC_MAX_SERVERS 16
+8 -6
View File
@@ -429,6 +429,8 @@ extern "C" {
GGML_OP_GLU,
GGML_OP_SLEEP,
GGML_OP_COUNT,
};
@@ -1314,12 +1316,6 @@ extern "C" {
// indirect matrix multiplication
// an id of -1 skips the slot and zeroes the matching row of the result
//
// [TAG_MUL_MAT_ID_DUPLICATE_ID]
// note: this op has undefined behaviour if a given `a` is referenced more than once for a given `b`
// (i.e. duplicate experts per token are not guaranteed to be supported in all backends)
// TODO: add backend support: https://github.com/ggml-org/llama.cpp/pull/29998#issuecomment-6153742533
// the CPU backend skips negative ids, their rows of the result are zero
GGML_API struct ggml_tensor * ggml_mul_mat_id(
struct ggml_context * ctx,
struct ggml_tensor * as,
@@ -2599,6 +2595,12 @@ extern "C" {
int n_tasks,
void * userdata);
// Busy-wait operator for scheduling and backend testing.
GGML_API struct ggml_tensor * ggml_sleep(
struct ggml_context * ctx,
struct ggml_tensor * a, // wait after this tensor has executed, return value is simply a full view of this tensor
int32_t us); // amount of time to wait in microseconds
// loss function
GGML_API struct ggml_tensor * ggml_cross_entropy_loss(
+6
View File
@@ -2501,6 +2501,9 @@ static bool ggml_backend_cann_supports_op(ggml_backend_dev_t dev, const ggml_ten
break;
case GGML_OP_SET_ROWS:
{
if (op->src[0]->type == GGML_TYPE_BF16) {
return false;
}
switch (op->type) {
case GGML_TYPE_F32:
case GGML_TYPE_F16:
@@ -2553,6 +2556,9 @@ static bool ggml_backend_cann_supports_op(ggml_backend_dev_t dev, const ggml_ten
if (((const int32_t *) op->op_params)[15] != 0) {
return false; // FIXME: support ggml_rope_set_offset
}
if (op->src[0]->type != GGML_TYPE_F32 && op->src[0]->type != GGML_TYPE_F16) {
return false;
}
if (op->src[0]->ne[0] > 896) {
return false;
}
+7 -1
View File
@@ -2034,6 +2034,10 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
{
ggml_compute_forward_fill(params, tensor);
} break;
case GGML_OP_SLEEP:
{
ggml_compute_forward_sleep(params, tensor);
} break;
case GGML_OP_FLASH_ATTN_EXT:
{
ggml_compute_forward_flash_attn_ext(params, tensor);
@@ -2306,6 +2310,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
case GGML_OP_REPEAT:
case GGML_OP_REPEAT_BACK:
case GGML_OP_LEAKY_RELU:
case GGML_OP_SLEEP:
{
n_tasks = 1;
} break;
@@ -2928,7 +2933,8 @@ struct ggml_cplan ggml_graph_plan(
} break;
case GGML_OP_SET_ROWS:
{
if (node->src[0]->type == GGML_TYPE_F16 && node->type != GGML_TYPE_F16) {
if ((node->src[0]->type == GGML_TYPE_F16 && node->type != GGML_TYPE_F16) ||
(node->src[0]->type == GGML_TYPE_BF16 && node->type != GGML_TYPE_BF16)) {
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
}
} break;
+48 -3
View File
@@ -2238,6 +2238,19 @@ void ggml_compute_forward_fill(const ggml_compute_params * params, ggml_tensor *
}
}
// ggml_compute_forward_sleep
void ggml_compute_forward_sleep(const ggml_compute_params * params, ggml_tensor * dst) {
if (params->ith != 0) {
return;
}
// same clock as ggml_time_us() callers; on macOS steady_clock is CLOCK_MONOTONIC_RAW, which can run faster
const int64_t t_end = ggml_time_us() + ggml_get_op_params_i32(dst, 0);
while (ggml_time_us() < t_end) {}
}
// ggml_compute_tri
template <typename T>
@@ -5518,7 +5531,7 @@ static void ggml_compute_forward_set_rows_impl(
assert(ne0 == nc);
assert(ne2 == ne02);
assert(ne3 == ne03);
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16);
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16);
assert(ne02 % ne11 == 0);
assert(ne03 % ne12 == 0);
@@ -5565,6 +5578,20 @@ static void ggml_compute_forward_set_rows_impl(
from_float(wdata,
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
}
} else if constexpr (std::is_same_v<src_t, ggml_bf16_t>) {
if (dst->type == GGML_TYPE_BF16) {
memcpy(
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3),
((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
rs);
} else {
float * wdata = (float *) params->wdata + (nc + CACHE_LINE_SIZE_F32) * ith;
ggml_bf16_to_fp32_row(
(const ggml_bf16_t *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
wdata, nc);
from_float(wdata,
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
}
} else {
GGML_ABORT("src0->type = %d (%s) not supported", src0->type, ggml_type_name(src0->type));
}
@@ -5601,6 +5628,16 @@ void ggml_compute_forward_set_rows(
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
}
} break;
case GGML_TYPE_BF16:
{
if (src1->type == GGML_TYPE_I64) {
ggml_compute_forward_set_rows_impl<ggml_bf16_t, int64_t>(params, dst);
} else if (src1->type == GGML_TYPE_I32) {
ggml_compute_forward_set_rows_impl<ggml_bf16_t, int32_t>(params, dst);
} else {
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
}
} break;
default:
{
GGML_ABORT("src0->type = %d (%s) not supported", src0->type, ggml_type_name(src0->type));
@@ -6289,7 +6326,7 @@ static void rotate_pairs(const int64_t n, const int64_t n_offset, const float *
}
}
template<typename T> //float or ggml_fp16_t
template<typename T> //float, ggml_fp16_t or ggml_bf16_t
static void ggml_compute_forward_rope_flt(
const ggml_compute_params * params,
ggml_tensor * dst,
@@ -6299,7 +6336,7 @@ static void ggml_compute_forward_rope_flt(
const ggml_tensor * src1 = dst->src[1];
const ggml_tensor * src2 = dst->src[2];
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16);
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16);
GGML_ASSERT(src1->type == GGML_TYPE_I32);
float freq_base, freq_scale, ext_factor, attn_factor, beta_fast, beta_slow;
@@ -6459,6 +6496,10 @@ void ggml_compute_forward_rope(
{
ggml_compute_forward_rope_flt<ggml_fp16_t>(params, dst, true);
} break;
case GGML_TYPE_BF16:
{
ggml_compute_forward_rope_flt<ggml_bf16_t>(params, dst, true);
} break;
case GGML_TYPE_F32:
{
ggml_compute_forward_rope_flt<float>(params, dst, true);
@@ -6483,6 +6524,10 @@ void ggml_compute_forward_rope_back(
{
ggml_compute_forward_rope_flt<ggml_fp16_t>(params, dst, false);
} break;
case GGML_TYPE_BF16:
{
ggml_compute_forward_rope_flt<ggml_bf16_t>(params, dst, false);
} break;
case GGML_TYPE_F32:
{
ggml_compute_forward_rope_flt<float>(params, dst, false);
+1
View File
@@ -84,6 +84,7 @@ void ggml_compute_forward_top_k(const struct ggml_compute_params * params, struc
void ggml_compute_forward_leaky_relu(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_tri(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_fill(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_sleep(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_flash_attn_ext(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_flash_attn_back(
const struct ggml_compute_params * params,
@@ -1,17 +1,17 @@
#include "batch-get.cuh"
#include "batch-copy.cuh"
#include "common.cuh"
#include "ggml-cpp.h"
#include <climits>
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
struct ggml_cuda_batch_get_descriptor {
struct ggml_cuda_batch_copy_descriptor {
const unsigned char * src;
unsigned char * dst;
size_t size;
};
static __global__ void ggml_cuda_batch_get_kernel(const ggml_cuda_batch_get_descriptor * copies, size_t count) {
static __global__ void ggml_cuda_batch_copy_kernel(const ggml_cuda_batch_copy_descriptor * copies, size_t count) {
// Blocks along y cooperate on each descriptor.
const size_t index = blockIdx.x;
if (index >= count) {
@@ -50,11 +50,11 @@ static __global__ void ggml_cuda_batch_get_kernel(const ggml_cuda_batch_get_desc
}
}
struct ggml_cuda_batch_get_mapping {
struct ggml_cuda_batch_copy_mapping {
void * host = nullptr;
unsigned char * device = nullptr;
ggml_cuda_batch_get_mapping(void * host, size_t size) {
ggml_cuda_batch_copy_mapping(void * host, size_t size) {
const cudaError_t err = cudaHostRegister(host, size, cudaHostRegisterMapped | cudaHostRegisterPortable);
if (err != cudaSuccess) {
(void) cudaGetLastError();
@@ -70,10 +70,10 @@ struct ggml_cuda_batch_get_mapping {
}
}
ggml_cuda_batch_get_mapping(const ggml_cuda_batch_get_mapping &) = delete;
ggml_cuda_batch_get_mapping & operator=(const ggml_cuda_batch_get_mapping &) = delete;
ggml_cuda_batch_copy_mapping(const ggml_cuda_batch_copy_mapping &) = delete;
ggml_cuda_batch_copy_mapping & operator=(const ggml_cuda_batch_copy_mapping &) = delete;
~ggml_cuda_batch_get_mapping() {
~ggml_cuda_batch_copy_mapping() {
if (host) {
// Release only the registration acquired by this object.
CUDA_CHECK(cudaHostUnregister(host));
@@ -82,11 +82,12 @@ struct ggml_cuda_batch_get_mapping {
};
#endif
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
static bool ggml_cuda_copy_tensor_n(int device, cudaMemcpyKind direction,
const ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
#if defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
GGML_UNUSED(device);
GGML_UNUSED(direction);
GGML_UNUSED(tensors);
GGML_UNUSED(data);
GGML_UNUSED(offsets);
@@ -94,14 +95,17 @@ bool ggml_cuda_get_tensor_n(int device,
GGML_UNUSED(n_tensors);
return false;
#else
const bool to_host = direction == cudaMemcpyDeviceToHost;
// Provisional dispatch values based on experiments.
constexpr size_t min_batch_reads = 128;
constexpr size_t max_gather_read_bytes = 128 * 1024;
// Collect non-empty CPU destination ranges.
constexpr size_t max_kernel_read_bytes = 128 * 1024;
// Collect destination ranges; repeated sources are allowed.
std::vector<std::pair<uintptr_t, uintptr_t>> ranges;
ranges.reserve(n_tensors);
size_t payload_bytes = 0;
// Number of non-empty reads whose size is at most max_gather_read_bytes.
uintptr_t begin = UINTPTR_MAX;
uintptr_t end = 0;
// Number of non-empty reads whose size is at most max_kernel_read_bytes.
size_t n_reads_small = 0;
for (size_t i = 0; i < n_tensors; ++i) {
if (sizes[i] == 0) {
@@ -111,15 +115,19 @@ bool ggml_cuda_get_tensor_n(int device,
GGML_ASSERT(address <= UINTPTR_MAX - sizes[i]);
GGML_ASSERT(payload_bytes <= SIZE_MAX - sizes[i]);
payload_bytes += sizes[i];
n_reads_small += sizes[i] <= max_gather_read_bytes;
ranges.emplace_back(address, address + sizes[i]);
n_reads_small += sizes[i] <= max_kernel_read_bytes;
begin = std::min(begin, address);
end = std::max(end, address + sizes[i]);
const uintptr_t destination = to_host ? address : (uintptr_t) tensors[i]->data + offsets[i];
GGML_ASSERT(destination <= UINTPTR_MAX - sizes[i]);
ranges.emplace_back(destination, destination + sizes[i]);
}
if (ranges.empty()) {
return true;
}
// Total number of non-empty reads.
const size_t n_reads = ranges.size();
if (n_reads > SIZE_MAX / sizeof(ggml_cuda_batch_get_descriptor) || n_reads > INT_MAX) {
if (n_reads > SIZE_MAX / sizeof(ggml_cuda_batch_copy_descriptor) || n_reads > INT_MAX) {
return false;
}
if (n_reads < 2 || (n_reads < min_batch_reads && n_reads_small == n_reads)) {
@@ -127,72 +135,85 @@ bool ggml_cuda_get_tensor_n(int device,
}
std::sort(ranges.begin(), ranges.end());
for (size_t i = 1; i < ranges.size(); ++i) {
// Preserve the ordered copy path when CPU destinations overlap.
// Preserve the ordered copy path when destinations overlap.
if (ranges[i].first < ranges[i - 1].second) {
return false;
}
}
// Find the CPU span to register, including gaps between destinations.
const uintptr_t begin = ranges.front().first;
const uintptr_t end = ranges.back().second;
if (end - begin - payload_bytes > payload_bytes) {
// Limit gaps in the CPU span without underflow when sources overlap.
if (end - begin > payload_bytes && end - begin - payload_bytes > payload_bytes) {
return false;
}
// Register and map the existing CPU destination so the kernel can write to it directly.
ggml_cuda_batch_get_mapping mapping((void *) begin, end - begin);
// Register and map the CPU span for direct kernel access.
ggml_cuda_batch_copy_mapping mapping((void *) begin, end - begin);
if (!mapping.host) {
return false;
}
// Reads selected for gather; zero means all reads use DMA.
size_t n_reads_gather = mapping.device && n_reads_small >= min_batch_reads ? n_reads_small : 0;
// Reads selected for the kernel; zero means all reads use DMA.
size_t n_reads_kernel = mapping.device && n_reads_small >= min_batch_reads ? n_reads_small : 0;
// Keep descriptor storage alive until the final stream synchronization.
constexpr size_t max_batch_descriptors = 128 * 1024;
const size_t bytes = std::min(n_reads_gather, max_batch_descriptors) * sizeof(ggml_cuda_batch_get_descriptor);
ggml_backend_buffer_ptr descriptors(n_reads_gather ? ggml_backend_buft_alloc_buffer(ggml_backend_cuda_buffer_type(device), bytes) : nullptr);
const size_t bytes = std::min(n_reads_kernel, max_batch_descriptors) * sizeof(ggml_cuda_batch_copy_descriptor);
ggml_backend_buffer_ptr descriptors(n_reads_kernel ? ggml_backend_buft_alloc_buffer(ggml_backend_cuda_buffer_type(device), bytes) : nullptr);
if (!descriptors) {
n_reads_gather = 0;
n_reads_kernel = 0;
}
const auto host_type = ggml_backend_cuda_host_buffer_type();
ggml_backend_buffer_ptr host_descriptors(n_reads_gather ? ggml_backend_buft_alloc_buffer(host_type, bytes) : nullptr);
ggml_backend_buffer_ptr host_descriptors(n_reads_kernel ? ggml_backend_buft_alloc_buffer(host_type, bytes) : nullptr);
if (!host_descriptors || ggml_backend_buffer_get_type(host_descriptors.get()) != host_type) {
n_reads_gather = 0;
n_reads_kernel = 0;
}
if (n_reads_gather) {
auto * copies = static_cast<ggml_cuda_batch_get_descriptor *>(ggml_backend_buffer_get_base(host_descriptors.get()));
auto * device_copies = static_cast<ggml_cuda_batch_get_descriptor *>(ggml_backend_buffer_get_base(descriptors.get()));
if (n_reads_kernel) {
auto * copies = static_cast<ggml_cuda_batch_copy_descriptor *>(ggml_backend_buffer_get_base(host_descriptors.get()));
auto * device_copies = static_cast<ggml_cuda_batch_copy_descriptor *>(ggml_backend_buffer_get_base(descriptors.get()));
const int threads = 256;
const size_t nsm = ggml_cuda_info().devices[device].nsm;
size_t i = 0;
for (size_t first = 0; first < n_reads_gather; first += max_batch_descriptors) {
const size_t n_batch = std::min(n_reads_gather - first, max_batch_descriptors);
for (size_t first = 0; first < n_reads_kernel; first += max_batch_descriptors) {
const size_t n_batch = std::min(n_reads_kernel - first, max_batch_descriptors);
size_t count = 0;
for (; i < n_tensors && count < n_batch; ++i) {
if (sizes[i] == 0 || sizes[i] > max_gather_read_bytes) {
if (sizes[i] == 0 || sizes[i] > max_kernel_read_bytes) {
continue;
}
copies[count++] = {(const unsigned char *) tensors[i]->data + offsets[i],
mapping.device + ((uintptr_t) data[i] - begin), sizes[i]};
auto * host = mapping.device + ((uintptr_t) data[i] - begin);
auto * tensor = (unsigned char *) tensors[i]->data + offsets[i];
copies[count++] = {to_host ? tensor : host, to_host ? host : tensor, sizes[i]};
}
const size_t blocks_per_copy = std::min<size_t>(UINT16_MAX, std::max<size_t>(1, (nsm + n_batch - 1) / n_batch));
const dim3 blocks((unsigned int) n_batch, (unsigned int) blocks_per_copy);
// Stream ordering keeps each kernel ahead of the next descriptor overwrite.
CUDA_CHECK(cudaMemcpyAsync(device_copies, copies, n_batch * sizeof(copies[0]), cudaMemcpyHostToDevice, cudaStreamPerThread));
ggml_cuda_batch_get_kernel<<<blocks, threads, 0, cudaStreamPerThread>>>(device_copies, n_batch);
ggml_cuda_batch_copy_kernel<<<blocks, threads, 0, cudaStreamPerThread>>>(device_copies, n_batch);
CUDA_CHECK(cudaGetLastError());
if (first + n_batch < n_reads_gather) {
if (first + n_batch < n_reads_kernel) {
// Wait before refilling the pinned host descriptor buffer.
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
}
}
}
// Keep the destination pinned for DMA too; overlapping destinations already declined above.
// Keep the CPU span pinned for DMA too; overlapping destinations already declined above.
for (size_t i = 0; i < n_tensors; ++i) {
if (sizes[i] && (!n_reads_gather || sizes[i] > max_gather_read_bytes)) {
CUDA_CHECK(cudaMemcpyAsync(data[i], (const char *) tensors[i]->data + offsets[i], sizes[i], cudaMemcpyDeviceToHost, cudaStreamPerThread));
if (sizes[i] && (!n_reads_kernel || sizes[i] > max_kernel_read_bytes)) {
void * host = const_cast<void *>(data[i]);
void * tensor = (char *) tensors[i]->data + offsets[i];
CUDA_CHECK(cudaMemcpyAsync(to_host ? host : tensor, to_host ? tensor : host, sizes[i], direction, cudaStreamPerThread));
}
}
// Wait for GPU writes before releasing the CPU mapping and descriptor buffers.
// Wait for transfers before releasing the CPU mapping and descriptor buffers.
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
return true;
#endif
}
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
return ggml_cuda_copy_tensor_n(device, cudaMemcpyDeviceToHost, tensors, data, offsets, sizes, n_tensors);
}
bool ggml_cuda_set_tensor_n(int device,
ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors) {
return ggml_cuda_copy_tensor_n(device, cudaMemcpyHostToDevice, tensors, data, offsets, sizes, n_tensors);
}
@@ -5,3 +5,7 @@
bool ggml_cuda_get_tensor_n(int device,
const ggml_tensor * const * tensors, void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors);
bool ggml_cuda_set_tensor_n(int device,
ggml_tensor * const * tensors, const void * const * data, const size_t * offsets,
const size_t * sizes, size_t n_tensors);
+10
View File
@@ -452,6 +452,16 @@ static __device__ __forceinline__ ggml_cuda_thread_mask_t ggml_cuda_ballot(int p
return __ballot_sync(mask, predicate);
}
static __device__ __forceinline__ int ggml_cuda_popc(ggml_cuda_thread_mask_t x) {
static_assert(sizeof(ggml_cuda_thread_mask_t) == 4 || sizeof(ggml_cuda_thread_mask_t) == 8, "unsupported thread mask type");
if constexpr (sizeof(ggml_cuda_thread_mask_t) == 4) {
return __popc(x);
} else {
return __popcll(x);
}
}
template<typename T> static __device__ __forceinline__ int ggml_cuda_popc(T x) = delete; // Makes implicit type casts result in an error.
static __device__ __forceinline__ int ggml_cuda_all(int predicate, ggml_cuda_thread_mask_t mask = GGML_CUDA_FULL_THREAD_MASK) {
return __all_sync(mask, predicate);
}
+7 -16
View File
@@ -17,15 +17,6 @@ static __global__ void flash_attn_mask_to_sparse_indices(
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
constexpr int values_per_lane = 8;
using lane_mask_t = std::conditional_t<warp_size == 64, uint64_t, uint32_t>;
constexpr uint64_t ballot_mask = ~(lane_mask_t)0;
const auto popc = [](const lane_mask_t mask) {
if constexpr (warp_size == 64) {
return __popcll(mask);
} else {
return __popc(mask);
}
};
const int tid = threadIdx.x;
const int warp = tid / warp_size;
const int lane = tid % warp_size;
@@ -48,7 +39,7 @@ static __global__ void flash_attn_mask_to_sparse_indices(
__syncthreads();
for (int i0 = 0; i0 < ne30; i0 += blockDim.x*values_per_lane) {
lane_mask_t selected_warp[values_per_lane];
ggml_cuda_thread_mask_t selected_warp[values_per_lane];
int warp_count = 0;
#pragma unroll
for (int item = 0; item < values_per_lane; ++item) {
@@ -60,8 +51,8 @@ static __global__ void flash_attn_mask_to_sparse_indices(
selected |= (!oob || q < q1 - q0) && isfinite(__half2float(mask[q*s31 + i]));
}
}
selected_warp[item] = (lane_mask_t)ggml_cuda_ballot((int) selected, ballot_mask);
warp_count += popc(selected_warp[item]);
selected_warp[item] = ggml_cuda_ballot((int) selected);
warp_count += ggml_cuda_popc(selected_warp[item]);
}
if (lane == 0) {
@@ -81,16 +72,16 @@ static __global__ void flash_attn_mask_to_sparse_indices(
}
__syncthreads();
const lane_mask_t lane_mask = lane == 0 ? 0 : ((lane_mask_t)1 << lane) - 1;
const ggml_cuda_thread_mask_t lane_mask = lane == 0 ? 0 : ((ggml_cuda_thread_mask_t)1 << lane) - 1;
int warp_item_offset = 0;
#pragma unroll
for (int item = 0; item < values_per_lane; ++item) {
const int i = i0 + (warp*values_per_lane + item)*warp_size + lane;
const int dst = row_count + warp_offsets[warp] + warp_item_offset + popc(selected_warp[item] & lane_mask);
if ((selected_warp[item] & ((lane_mask_t)1 << lane)) && dst < n_kv_max) {
const int dst = row_count + warp_offsets[warp] + warp_item_offset + ggml_cuda_popc(selected_warp[item] & lane_mask);
if ((selected_warp[item] & ((ggml_cuda_thread_mask_t)1 << lane)) && dst < n_kv_max) {
indices[dst] = i;
}
warp_item_offset += popc(selected_warp[item]);
warp_item_offset += ggml_cuda_popc(selected_warp[item]);
}
__syncthreads();
+28 -5
View File
@@ -10,7 +10,7 @@
#include "ggml-cuda/arange.cuh"
#include "ggml-cuda/argmax.cuh"
#include "ggml-cuda/argsort.cuh"
#include "ggml-cuda/batch-get.cuh"
#include "ggml-cuda/batch-copy.cuh"
#include "ggml-cuda/binbcast.cuh"
#include "ggml-cuda/clamp.cuh"
#include "ggml-cuda/col2im-1d.cuh"
@@ -48,6 +48,7 @@
#include "ggml-cuda/rope.cuh"
#include "ggml-cuda/roll.cuh"
#include "ggml-cuda/scale.cuh"
#include "ggml-cuda/sleep.cuh"
#include "ggml-cuda/snake.cuh"
#include "ggml-cuda/softcap.cuh"
#include "ggml-cuda/softmax.cuh"
@@ -1116,6 +1117,11 @@ static void ggml_backend_cuda_buffer_set_tensor_n(ggml_backend_buffer_t buffer,
ggml_cuda_set_device(ctx->device);
if (!ggml_cuda_pinned_memory_disabled() &&
ggml_cuda_set_tensor_n(ctx->device, tensors, data, offsets, sizes, n_tensors)) {
return;
}
for (size_t i = 0; i < n_tensors; i++) {
if (sizes[i] == 0) {
continue;
@@ -2724,7 +2730,8 @@ static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor *
ids_from_sorted_host[i12*n_expert_used + iex] = ids_to_sorted_host.size();
ids_to_sorted_host.push_back(i12*ne11 + iex % ne11);
tokens_per_expert[i02]++;
break;
// TODO: removing the break here was done to make duplicate IDs work but this caused a ~10% performance regression, better solution needed.
// break;
}
}
}
@@ -3145,6 +3152,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
case GGML_OP_FILL:
ggml_cuda_op_fill(ctx, dst);
break;
case GGML_OP_SLEEP:
ggml_cuda_op_sleep(ctx, dst);
break;
case GGML_OP_LIGHTNING_INDEXER:
ggml_cuda_lightning_indexer(ctx, dst);
break;
@@ -3495,7 +3505,12 @@ static bool ggml_cuda_should_fuse_rope_set_rows(const ggml_tensor * rope,
return false;
}
if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16) {
if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16 && set_rows->type != GGML_TYPE_BF16) {
return false;
}
// BF16 rope fuses into an F16 or BF16 cache, other ropes not into a BF16 cache
if (rope->type == GGML_TYPE_BF16 ? set_rows->type == GGML_TYPE_F32 : set_rows->type == GGML_TYPE_BF16) {
return false;
}
@@ -6501,7 +6516,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
(op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_BF16 || op->type == GGML_TYPE_F8_E4M3 ||
op->type == GGML_TYPE_Q4_0 || op->type == GGML_TYPE_Q4_1 || op->type == GGML_TYPE_Q5_0 ||
op->type == GGML_TYPE_Q5_1 || op->type == GGML_TYPE_Q8_0 || op->type == GGML_TYPE_IQ4_NL) &&
op->src[0]->type == GGML_TYPE_F32
(op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_BF16)
) || (
op->type == GGML_TYPE_F16 && op->src[0]->type == GGML_TYPE_F16
)
@@ -6722,7 +6737,9 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
return false;
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK: {
return op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]);
return (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16 || op->src[0]->type == GGML_TYPE_BF16) &&
op->type == op->src[0]->type &&
op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]);
}
case GGML_OP_IM2COL:
case GGML_OP_IM2COL_3D:
@@ -6809,6 +6826,12 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
return ggml_cuda_lightning_indexer_supported(dev_ctx->device, op);
case GGML_OP_TOP_K_MASK:
return op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_F32;
case GGML_OP_SLEEP:
#if defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
return false;
#else
return true;
#endif // defined(GGML_USE_HIP) || defined(GGML_USE_MUSA)
default:
return false;
+4 -5
View File
@@ -43,7 +43,6 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
const int64_t s13 = src1->nb[3] / ts_src1;
const int64_t s3 = dst->nb[3] / ts_dst;
const int64_t ids_s0 = ids ? ids->nb[0] / ggml_type_size(ids->type) : 0;
const int64_t ids_s1 = ids ? ids->nb[1] / ggml_type_size(ids->type) : 0;
mmf_ids_data ids_info{};
@@ -70,7 +69,7 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
nchannels_y = ids->ne[0];
}
if (ids && ncols_dst > 16) {
if (ids) {
const int64_t n_expert_used = ids->ne[0];
const int64_t n_experts = ne02;
const int64_t n_tokens = ne12;
@@ -110,7 +109,7 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
constexpr int vals_per_T = 1;
mul_mat_f_switch_rows_per_block<float>(
rows_per_block, src0_d, src1_d, ids_d, dst_d, ne00/vals_per_T, ne01, ncols_dst, s01/vals_per_T, stride_col_y/vals_per_T, stride_col_dst,
ids_s0, ids_s1, ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne03, ne3, s03/vals_per_T, s13, s3, ctx.stream(), ids_info_ptr);
} break;
case GGML_TYPE_F16: {
@@ -118,7 +117,7 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
constexpr int vals_per_T = 2;
mul_mat_f_switch_rows_per_block<half2>(
rows_per_block, src0_d, src1_d, ids_d, dst_d, ne00/vals_per_T, ne01, ncols_dst, s01/vals_per_T, stride_col_y/vals_per_T, stride_col_dst,
ids_s0, ids_s1, ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne03, ne3, s03/vals_per_T, s13, s3, ctx.stream(), ids_info_ptr);
} break;
case GGML_TYPE_BF16: {
@@ -126,7 +125,7 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
constexpr int vals_per_T = 2;
mul_mat_f_switch_rows_per_block<nv_bfloat162>(
rows_per_block, src0_d, src1_d, ids_d, dst_d, ne00/vals_per_T, ne01, ncols_dst, s01/vals_per_T, stride_col_y/vals_per_T, stride_col_dst,
ids_s0, ids_s1, ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne02, nchannels_y, nchannels_dst, s02/vals_per_T, stride_channel_y, stride_channel_dst,
ne03, ne3, s03/vals_per_T, s13, s3, ctx.stream(), ids_info_ptr);
} break;
default:
+69 -176
View File
@@ -47,12 +47,11 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
bool ggml_cuda_should_use_mmf(enum ggml_type type, int cc, int warp_size, const int64_t * scr0_ne, const size_t * src0_nb, const int src1_ncols, bool mul_mat_id);
template <typename T, int rows_per_block, int cols_per_block, int nwarps, bool has_ids>
template <typename T, int rows_per_block, int cols_per_block, int nwarps>
__launch_bounds__(ggml_cuda_get_physical_warp_size()*nwarps, 1)
static __global__ void mul_mat_f(
const T * __restrict__ x, const float * __restrict__ y, const int32_t * __restrict__ ids, float * __restrict__ dst,
const int ncols, const int ncols_dst_total, const int nchannels_dst, const int stride_row, const int stride_col_y, const int stride_col_dst,
const int stride_col_id, const int stride_row_id,
const T * __restrict__ x, const float * __restrict__ y, float * __restrict__ dst,
const int ncols, const int stride_row, const int stride_col_y, const int stride_col_dst,
const int channel_ratio, const int stride_channel_x, const int stride_channel_y, const int stride_channel_dst,
const int sample_ratio, const int stride_sample_x, const int stride_sample_y, const int stride_sample_dst) {
// TODO: handle this in a consistent and simpler way after AMD MFMA support has been added
@@ -90,85 +89,26 @@ static __global__ void mul_mat_f(
constexpr int ntA = rows_per_block / tile_A::I;
constexpr int ntB = (cols_per_block + tile_B::I - 1) / tile_B::I;
const int row0 = blockIdx.x * rows_per_block;
const int row0 = blockIdx.x * rows_per_block;
int expert_idx = 0;
[[maybe_unused]] int col_base = 0;
const int channel_dst = has_ids ? 0 : blockIdx.y;
if constexpr (has_ids) {
// experts + tiles of ncols_dst are packed in the y dimension
int col_tiles = (ncols_dst_total + cols_per_block - 1) / cols_per_block;
const int nchannels_x = gridDim.y / col_tiles;
const int tile_idx = blockIdx.y / nchannels_x;
expert_idx = blockIdx.y - tile_idx * nchannels_x;
col_base = tile_idx * cols_per_block;
}
const int channel_x = has_ids ? expert_idx : (channel_dst / channel_ratio);
const int channel_dst = blockIdx.y;
const int channel_x = channel_dst / channel_ratio;
const int channel_y = channel_dst;
const int sample_dst = blockIdx.z;
const int sample_x = sample_dst / sample_ratio;
const int sample_y = sample_dst;
x += int64_t(sample_x) *stride_sample_x + channel_x *stride_channel_x + row0*stride_row ;
y += int64_t(sample_y) *stride_sample_y + (has_ids ? 0 : channel_y *stride_channel_y);
dst += int64_t(sample_dst)*stride_sample_dst + (has_ids ? 0 : channel_dst*stride_channel_dst);
if constexpr (has_ids) {
constexpr int y_stride_scale = std::is_same_v<T, float> ? 1 : 2;
const int64_t col_offset = col_base;
y += col_offset * stride_col_y * y_stride_scale;
dst += col_offset * stride_col_dst;
ids += col_offset * stride_row_id;
}
y += int64_t(sample_y) *stride_sample_y + channel_y *stride_channel_y;
dst += int64_t(sample_dst)*stride_sample_dst + channel_dst*stride_channel_dst;
[[maybe_unused]] const float2 * y2 = (const float2 *) y;
extern __shared__ char data_mmv[];
char * shmem_base = data_mmv;
[[maybe_unused]] int * slot_map = (int *) shmem_base;
char * compute_base = has_ids ? (shmem_base + GGML_PAD(cols_per_block, 16) * sizeof(int)) : shmem_base;
tile_C C[ntA][ntB];
T * tile_xy = (T *) compute_base + threadIdx.y*(tile_A::I * tile_k_padded);
if constexpr (has_ids) {
int found = 0;
for (int j0 = 0; j0 < cols_per_block; j0 += nwarps) {
const int j = j0 + threadIdx.y;
if (threadIdx.x == 0) {
slot_map[j] = -1;
}
ggml_cuda_syncwarp();
if (col_base + j >= ncols_dst_total) {
continue;
}
const int32_t * __restrict__ id_row = ids + j*stride_row_id;
for (int k = threadIdx.x; k < nchannels_dst; k += warp_size) {
int match = id_row[k*stride_col_id] == expert_idx;
if (match) {
slot_map[j] = k;
found = 1;
break;
}
}
}
if (!__syncthreads_or(found)) {
return;
}
}
T * tile_xy = (T *) data_mmv + threadIdx.y*(tile_A::I * tile_k_padded);
for (int col = threadIdx.y*warp_size + threadIdx.x; col < ncols; col += nwarps*warp_size) {
tile_A A[ntA][warp_size / tile_A::J];
@@ -194,26 +134,15 @@ static __global__ void mul_mat_f(
for (int j0 = 0; j0 < tile_B::I; ++j0) {
const int j = j0 + itB*tile_B::I;
if constexpr (!has_ids) {
tile_xy[j0*tile_k_padded + threadIdx.x] = j < cols_per_block ? y[j*stride_col_y + col] : 0.0f;
} else {
const bool valid = j < cols_per_block && (col_base + j) < ncols_dst_total && slot_map[j] >= 0;
tile_xy[j0*tile_k_padded + threadIdx.x] = valid ? y[slot_map[j]*stride_channel_y + j*stride_col_y + col] : 0.0f;
}
tile_xy[j0*tile_k_padded + threadIdx.x] = j < cols_per_block ? y[j*stride_col_y + col] : 0.0f;
}
} else if constexpr (std::is_same_v<T, half2> || std::is_same_v<T, nv_bfloat162>) {
#pragma unroll
for (int j0 = 0; j0 < tile_B::I; ++j0) {
const int j = j0 + itB*tile_B::I;
if constexpr (!has_ids) {
const float2 tmp = j < cols_per_block ? y2[j*stride_col_y + col] : make_float2(0.0f, 0.0f);
tile_xy[j0*tile_k_padded + threadIdx.x] = ggml_cuda_cast<T>(tmp);
} else {
const bool valid = j < cols_per_block && (col_base + j) < ncols_dst_total && slot_map[j] >= 0;
float2 tmp = valid ? *(const float2*) &y[slot_map[j]*stride_channel_y + 2*(j*stride_col_y + col)] : make_float2(0.0f, 0.0f);
tile_xy[j0*tile_k_padded + threadIdx.x] = ggml_cuda_cast<T>(tmp);
}
const float2 tmp = j < cols_per_block ? y2[j*stride_col_y + col] : make_float2(0.0f, 0.0f);
tile_xy[j0*tile_k_padded + threadIdx.x] = ggml_cuda_cast<T>(tmp);
}
} else {
static_assert(std::is_same_v<T, void>, "unsupported type");
@@ -231,7 +160,7 @@ static __global__ void mul_mat_f(
}
}
float * buf_iw = (float *) compute_base;
float * buf_iw = (float *) data_mmv;
constexpr int kiw = nwarps*rows_per_block + mmf_get_padding();
if (nwarps > 1) {
@@ -278,26 +207,15 @@ static __global__ void mul_mat_f(
}
}
if constexpr (!has_ids) {
#pragma unroll
for (int i0 = 0; i0 < sizeof(sum)/sizeof(sum[0]); ++i0) {
dst[j*stride_col_dst + row0 + i0*warp_size + threadIdx.x] = sum[i0];
}
} else {
const int slot = (j < cols_per_block) ? slot_map[j] : -1;
if (slot >= 0 && (col_base + j) < ncols_dst_total) {
#pragma unroll
for (int i0 = 0; i0 < sizeof(sum)/sizeof(sum[0]); ++i0) {
dst[slot*stride_channel_dst + j*stride_col_dst + row0 + i0*warp_size + threadIdx.x] = sum[i0];
}
}
for (int i0 = 0; i0 < sizeof(sum)/sizeof(sum[0]); ++i0) {
dst[j*stride_col_dst + row0 + i0*warp_size + threadIdx.x] = sum[i0];
}
}
}
#else
GGML_UNUSED_VARS(x, y, ids, dst,
ncols, ncols_dst_total, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id,
GGML_UNUSED_VARS(x, y, dst,
ncols, stride_row, stride_col_y, stride_col_dst,
channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst);
NO_DEVICE_CODE;
@@ -588,22 +506,21 @@ static __global__ void mul_mat_f_ids(
template<typename T, int rows_per_block, int cols_per_block, int nwarps>
static inline void mul_mat_f_switch_ids(
const T * x, const float * y, const int32_t * ids, float * dst,
const T * x, const float * y, float * dst,
const int64_t ncols_x, const int64_t ncols_dst, const int64_t nchannels_dst,
const int64_t stride_row, const int64_t stride_col_y, const int64_t stride_col_dst,
const int64_t stride_col_id, const int64_t stride_row_id,
const int64_t channel_ratio, const int64_t stride_channel_x, const int64_t stride_channel_y, const int64_t stride_channel_dst,
const int64_t sample_ratio, const int64_t stride_sample_x, const int64_t stride_sample_y, const int64_t stride_sample_dst,
const dim3 & block_nums, const dim3 & block_dims, const int nbytes_shared_total, cudaStream_t stream,
const mmf_ids_data * ids_data) {
const bool has_ids_data = ids_data && ids_data->ids_src_compact;
// Use the compact-ids kernel only for larger tiles; for small ncols_dst (< 16)
// we prefer the normal mul_mat_f path with has_ids=true.
if (has_ids_data && ncols_dst > 16) {
if (has_ids_data) {
if (ids_data->max_tiles == 0) {
return;
}
GGML_ASSERT(block_nums.z == 1);
dim3 block_nums_ids(block_nums.x, ids_data->max_tiles, 1);
const uint3 sis1_fd = ids_data->sis1 > 0 ? init_fastdiv_values((uint32_t) ids_data->sis1) : make_uint3(0, 0, 1);
@@ -615,19 +532,10 @@ static inline void mul_mat_f_switch_ids(
channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
sis1_fd, nch_fd);
} else if (ids) {
const int64_t col_tiles = (ncols_dst + cols_per_block - 1) / cols_per_block;
dim3 block_nums_ids = block_nums;
block_nums_ids.y *= col_tiles;
mul_mat_f<T, rows_per_block, cols_per_block, nwarps, true><<<block_nums_ids, block_dims, nbytes_shared_total, stream>>>
(x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst);
} else {
mul_mat_f<T, rows_per_block, cols_per_block, nwarps, false><<<block_nums, block_dims, nbytes_shared_total, stream>>>
(x, y, ids, dst, ncols_x, cols_per_block, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
mul_mat_f<T, rows_per_block, cols_per_block, nwarps><<<block_nums, block_dims, nbytes_shared_total, stream>>>
(x, y, dst, ncols_x, stride_row, stride_col_y, stride_col_dst,
channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst);
}
}
@@ -637,7 +545,6 @@ void mul_mat_f_cuda(
const T * x, const float * y, const int32_t * ids, float * dst,
const int64_t ncols_x, const int64_t nrows_x, const int64_t ncols_dst,
const int64_t stride_row, const int64_t stride_col_y, const int64_t stride_col_dst,
const int64_t stride_col_id, const int64_t stride_row_id,
const int64_t nchannels_x, const int64_t nchannels_y, const int64_t nchannels_dst,
const int64_t stride_channel_x, const int64_t stride_channel_y, const int64_t stride_channel_dst, const int64_t nsamples_x,
const int64_t nsamples_dst, const int64_t stride_sample_x, const int64_t stride_sample_y, const int64_t stride_sample_dst,
@@ -673,70 +580,59 @@ void mul_mat_f_cuda(
const int nbytes_shared_iter = nwarps_best * (volta_mma_available(cc) ? tile_A_32::I : tile_A_16::I) * (warp_size + mmf_get_padding(cc)) * 4;
const int nbytes_cols_per_block_pad = (amd_wmma_available(cc) || amd_mfma_available(cc)) ? tile_B_16::I : tile_B_8::I;
const int nbytes_shared_combine = GGML_PAD(cols_per_block, nbytes_cols_per_block_pad) * (nwarps_best*rows_per_block + mmf_get_padding(cc)) * 4;
const int nbytes_shared = std::max(nbytes_shared_iter, nbytes_shared_combine);
const int nbytes_slotmap = ids ? GGML_PAD(cols_per_block, 16) * sizeof(int) : 0;
const int nbytes_shared_total = nbytes_shared + nbytes_slotmap;
const int64_t grid_y = ids ? nchannels_x : nchannels_dst;
const int nbytes_shared_total = std::max(nbytes_shared_iter, nbytes_shared_combine);
const dim3 block_nums(nrows_x/rows_per_block, grid_y, nsamples_dst);
const dim3 block_nums(nrows_x/rows_per_block, nchannels_dst, nsamples_dst);
const dim3 block_dims(warp_size, nwarps_best, 1);
switch (nwarps_best) {
case 1: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 1>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 2: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 2>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 3: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 3>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 4: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 4>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 5: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 5>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 6: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 6>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 7: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 7>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
case 8: {
mul_mat_f_switch_ids<T, rows_per_block, cols_per_block, 8>(
x, y, ids, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, channel_ratio, stride_channel_x, stride_channel_y, stride_channel_dst,
sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst, block_nums, block_dims, nbytes_shared_total, stream,
ids_data);
x, y, dst, ncols_x, ncols_dst, nchannels_dst, stride_row, stride_col_y, stride_col_dst, channel_ratio,
stride_channel_x, stride_channel_y, stride_channel_dst, sample_ratio, stride_sample_x, stride_sample_y, stride_sample_dst,
block_nums, block_dims, nbytes_shared_total, stream, ids_data);
} break;
default:
GGML_ABORT("fatal error");
@@ -750,95 +646,94 @@ static void mul_mat_f_switch_cols_per_block(
const T * x, const float * y, const int32_t * ids, float * dst,
const int64_t ncols_x, const int64_t nrows_x, const int64_t ncols_dst,
const int64_t stride_row, const int64_t stride_col_y, const int64_t stride_col_dst,
const int64_t stride_col_id, const int stride_row_id,
const int64_t nchannels_x, const int64_t nchannels_y, const int64_t nchannels_dst,
const int64_t stride_channel_x, const int64_t stride_channel_y, const int64_t stride_channel_dst, const int64_t nsamples_x,
const int64_t nsamples_dst, const int64_t stride_sample_x, const int64_t stride_sample_y, const int64_t stride_sample_dst,
cudaStream_t stream, const mmf_ids_data * ids_data) {
const int ncols_case = (ids && ncols_dst > 16) ? 16 : ncols_dst;
const int ncols_case = ids ? 16 : ncols_dst;
GGML_ASSERT(ids || ncols_dst <= 16);
switch (ncols_case) {
case 1: {
mul_mat_f_cuda<T, rows_per_block, 1>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 2: {
mul_mat_f_cuda<T, rows_per_block, 2>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 3: {
mul_mat_f_cuda<T, rows_per_block, 3>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 4: {
mul_mat_f_cuda<T, rows_per_block, 4>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 5: {
mul_mat_f_cuda<T, rows_per_block, 5>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 6: {
mul_mat_f_cuda<T, rows_per_block, 6>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 7: {
mul_mat_f_cuda<T, rows_per_block, 7>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 8: {
mul_mat_f_cuda<T, rows_per_block, 8>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 9: {
mul_mat_f_cuda<T, rows_per_block, 9>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 10: {
mul_mat_f_cuda<T, rows_per_block, 10>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 11: {
mul_mat_f_cuda<T, rows_per_block, 11>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 12: {
mul_mat_f_cuda<T, rows_per_block, 12>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 13: {
mul_mat_f_cuda<T, rows_per_block, 13>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 14: {
mul_mat_f_cuda<T, rows_per_block, 14>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 15: {
mul_mat_f_cuda<T, rows_per_block, 15>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case 16: {
mul_mat_f_cuda<T, rows_per_block, 16>(x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
default:
@@ -851,7 +746,6 @@ static void mul_mat_f_switch_rows_per_block(
const int rows_per_block, const T * x, const float * y, const int32_t * ids, float * dst,
const int64_t ncols_x, const int64_t nrows_x, const int64_t ncols_dst,
const int64_t stride_row, const int64_t stride_col_y, const int64_t stride_col_dst,
const int64_t stride_col_id, const int stride_row_id,
const int64_t nchannels_x, const int64_t nchannels_y, const int64_t nchannels_dst,
const int64_t stride_channel_x, const int64_t stride_channel_y, const int64_t stride_channel_dst, const int64_t nsamples_x,
const int64_t nsamples_dst, const int64_t stride_sample_x, const int64_t stride_sample_y, const int64_t stride_sample_dst,
@@ -860,13 +754,13 @@ static void mul_mat_f_switch_rows_per_block(
case MMF_ROWS_PER_BLOCK: {
mul_mat_f_switch_cols_per_block<T, MMF_ROWS_PER_BLOCK>(
x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
case MMF_ROWS_PER_BLOCK_CDNA: {
mul_mat_f_switch_cols_per_block<T, MMF_ROWS_PER_BLOCK_CDNA>(
x, y, ids, dst, ncols_x, nrows_x, ncols_dst, stride_row, stride_col_y, stride_col_dst,
stride_col_id, stride_row_id, nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, stream, ids_data);
} break;
default:
@@ -878,7 +772,6 @@ static void mul_mat_f_switch_rows_per_block(
template void mul_mat_f_cuda<T, nrows_dst, ncols_dst>( \
const T * x, const float * y, const int32_t * ids, float * dst, \
const int64_t ncols_x, const int64_t nrows_x, int64_t ncols_dst_total, const int64_t stride_row, const int64_t stride_col_y, const int64_t stride_col_dst, \
const int64_t stride_col_id, const int64_t stride_row_id, \
const int64_t nchannels_x, const int64_t nchannels_y, const int64_t nchannels_dst, \
const int64_t stride_channel_x, const int64_t stride_channel_y, const int64_t stride_channel_dst, const int64_t nsamples_x,\
const int64_t nsamples_dst, const int64_t stride_sample_x, const int64_t stride_sample_y, const int64_t stride_sample_dst, \
+159 -60
View File
@@ -24,6 +24,50 @@ template <int n> struct mm_ids_pow2 { static constexpr int value = 2*mm_ids_pow2
template <> struct mm_ids_pow2<1> { static constexpr int value = 1; };
template <> struct mm_ids_pow2<0> { static constexpr int value = 1; };
template <bool increment_nex_prev, bool write_to_store>
static __device__ __forceinline__ void check_experts(
mm_ids_helper_store * store, int & nex_prev, int & it_compact,
const int expert, const int it, const int iex, const int expert_found) {
if (increment_nex_prev) {
nex_prev += expert_found < expert;
}
if (write_to_store) {
const ggml_cuda_thread_mask_t iex_used = ggml_cuda_ballot(expert_found == expert);
const ggml_cuda_thread_mask_t mask = (ggml_cuda_thread_mask_t(1) << threadIdx.x) - 1;
const int it_compact_offset = ggml_cuda_popc(iex_used & mask);
if (expert_found == expert) {
store[it_compact + it_compact_offset] = mm_ids_helper_store(it, iex);
}
it_compact += ggml_cuda_popc(iex_used);
}
}
static __device__ __forceinline__ void dump_store(
int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst,
const int n_expert_used, const int nchannels_y, const int sis1, const bool write_inverse,
const mm_ids_helper_store * __restrict__ store, const int nex_prev, int & n_dumped, const int it_compact) {
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
for (int itc = threadIdx.x; itc < it_compact; itc += warp_size) {
const mm_ids_helper_store store_it = store[itc];
const int it = store_it.it();
const int iex_used = store_it.iex_used();
ids_dst[nex_prev + n_dumped + itc] = it*n_expert_used + iex_used;
// ids_src1 holds the forward map, or the inverse map (token slot -> compact row) for quant dedup
if (write_inverse) {
ids_src1[it*n_expert_used + iex_used] = nex_prev + n_dumped + itc;
} else {
ids_src1[nex_prev + n_dumped + itc] = it*sis1 + iex_used % nchannels_y;
}
}
n_dumped += it_compact;
}
constexpr int n_unroll = 4; // Manually unroll this many loop iterations for better performance.
// Helper function for mul_mat_id, converts ids to a more convenient format.
// ids_src1 describes how to permute the flattened column indices of src1 in order to get a compact src1 tensor sorted by expert.
// ids_dst describes the same mapping but for the dst tensor.
@@ -36,85 +80,138 @@ static __global__ void mm_ids_helper(
const int32_t * __restrict__ ids,
int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, ggml_cuda_mm_id_tile_spec * __restrict__ tile_spec,
const int n_tokens, const int n_expert_used_var, const int nchannels_y, const uint3 tile_width,
const int si1, const int sis1, const bool write_inverse) {
const int si1, const int sis1, const bool write_inverse, const int ne_store) {
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
const int n_expert_used = n_expert_used_template == 0 ? n_expert_used_var : n_expert_used_template;
const int expert = blockIdx.x;
const int expert = blockIdx.x; // The expert this CUDA block handles.
extern __shared__ char data_mm_ids_helper[];
mm_ids_helper_store * store = (mm_ids_helper_store *) data_mm_ids_helper;
int nex_prev = 0; // Number of columns for experts with a lower index.
int n_dumped = 0; // Number of data values that have been dumped to output buffers.
int it_compact = 0; // Running index for the compact slice of this expert.
// Step 1: Iterate over n_tokens, increment nex_prev to determine the output indices, fill up store.
// Step 1.5: If the store does not have enough space to hold all values, iterate over the rest of the tokens just to increment nex_prev.
// Step 2: Dump the stored values using nex_prev to get the correct output indices.
// Step 3: If we were not able to process all values, resume at the index up to which we were able to dump the output and re-do.
if constexpr (n_expert_used_template == 0) {
// Generic implementation:
for (int it = 0; it < n_tokens; ++it) {
int iex_used = -1; // The index at which the expert is used, if any.
for (int iex = threadIdx.x; iex < n_expert_used; iex += warp_size) {
const int expert_used = ids[it*si1 + iex];
nex_prev += expert_used < expert;
if (expert_used == expert) {
iex_used = iex;
int it = 0;
for (; it < n_tokens && it_compact + max(warp_size, n_expert_used) <= ne_store; ++it) {
for (int iex0 = 0; iex0 < n_expert_used; iex0 += warp_size) {
const int iex = iex0 + threadIdx.x;
const int expert_found = iex < n_expert_used ? ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = true;
constexpr bool write_to_store = true;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
}
}
int it_dumped = it;
for (; it < n_tokens; ++it) {
for (int iex0 = 0; iex0 < n_expert_used; iex0 += warp_size) {
const int iex = iex0 + threadIdx.x;
const int expert_found = iex < n_expert_used ? ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = true;
constexpr bool write_to_store = false;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
}
}
nex_prev = warp_reduce_sum<warp_size>(nex_prev);
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
it_compact = 0;
for (int it = it_dumped; it < n_tokens; ++it) {
for (int iex0 = 0; iex0 < n_expert_used; iex0 += warp_size) {
const int iex = iex0 + threadIdx.x;
const int expert_found = iex < n_expert_used ? ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = false;
constexpr bool write_to_store = true;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
if (it_compact + max(warp_size, n_expert_used) >= ne_store) {
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
it_compact = 0;
}
}
if (iex_used != -1) {
store[it_compact] = mm_ids_helper_store(it, iex_used);
}
if (warp_reduce_any<warp_size>(iex_used != -1)) {
it_compact++;
}
}
if (it_compact > 0) {
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
}
} else {
// Implementation optimized for specific numbers of experts used:
// a warp holds a whole number of token slots, so the slot count is padded to a power of 2
constexpr int neu_padded = mm_ids_pow2<n_expert_used_template>::value;
static_assert(neu_padded <= warp_size && warp_size % neu_padded == 0, "bad n_expert_used");
for (int it0 = 0; it0 < n_tokens; it0 += warp_size/neu_padded) {
const int it = it0 + threadIdx.x / neu_padded;
const int iex = threadIdx.x % neu_padded; // The index at which the expert is used, if any.
const int iex = threadIdx.x % neu_padded; // The index at which the expert is used, if any.
const int expert_used = (neu_padded == n_expert_used || iex < n_expert_used) && it < n_tokens ?
ids[it*si1 + iex] : INT_MAX;
const int iex_used = expert_used == expert ? iex : -1;
nex_prev += expert_used < expert;
// Whether the threads at this token position have used the expert:
const int it_compact_add_self = warp_reduce_any<neu_padded>(iex_used != -1);
// Do a scan over threads at lower token positions in warp to get the correct index for writing data:
int it_compact_add_lower = 0;
int it0 = 0;
for (; it0 < n_tokens && it_compact + n_unroll*warp_size <= ne_store; it0 += n_unroll*warp_size/neu_padded) {
#pragma unroll
for (int offset = neu_padded; offset < warp_size; offset += neu_padded) {
const int tmp = ggml_cuda_shfl_up(it_compact_add_self, offset, warp_size);
if (threadIdx.x >= static_cast<unsigned int>(offset)) {
it_compact_add_lower += tmp;
for (int it1 = 0; it1 < n_unroll*warp_size/neu_padded; it1 += warp_size/neu_padded) {
const int it = it0 + it1 + threadIdx.x / neu_padded;
const int expert_found = (neu_padded == n_expert_used || iex < n_expert_used) && it < n_tokens ?
ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = true;
constexpr bool write_to_store = true;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
}
}
int it0_dumped = it0;
for (; it0 < n_tokens; it0 += n_unroll*warp_size/neu_padded) {
#pragma unroll
for (int it1 = 0; it1 < n_unroll*warp_size/neu_padded; it1 += warp_size/neu_padded) {
const int it = it0 + it1 + threadIdx.x / neu_padded;
const int expert_found = (neu_padded == n_expert_used || iex < n_expert_used) && it < n_tokens ?
ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = true;
constexpr bool write_to_store = false;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
}
}
nex_prev = warp_reduce_sum<warp_size>(nex_prev);
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
it_compact = 0;
for (it0 = it0_dumped; it0 < n_tokens; it0 += n_unroll*warp_size/neu_padded) {
#pragma unroll
for (int it1 = 0; it1 < n_unroll*warp_size/neu_padded; it1 += warp_size/neu_padded) {
const int it = it0 + it1 + threadIdx.x / neu_padded;
const int expert_found = (neu_padded == n_expert_used || iex < n_expert_used) && it < n_tokens ?
ids[it*si1 + iex] : INT_MAX;
constexpr bool increment_nex_prev = false;
constexpr bool write_to_store = true;
check_experts<increment_nex_prev, write_to_store>
(store, nex_prev, it_compact, expert, it, iex, expert_found);
if (it_compact + warp_size >= ne_store) {
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
it_compact = 0;
}
}
if (iex_used != -1) {
store[it_compact + it_compact_add_lower] = mm_ids_helper_store(it, iex_used);
}
// The thread with the highest index in the warp always has the sum over the whole warp, use it to increment all threads:
it_compact += ggml_cuda_shfl(it_compact_add_lower + it_compact_add_self, warp_size - 1, warp_size);
}
}
nex_prev = warp_reduce_sum<warp_size>(nex_prev);
ggml_cuda_syncwarp();
for (int itc = threadIdx.x; itc < it_compact; itc += warp_size) {
const mm_ids_helper_store store_it = store[itc];
const int it = store_it.it();
const int iex_used = store_it.iex_used();
ids_dst[nex_prev + itc] = it*n_expert_used + iex_used;
// ids_src1 holds the forward map, or the inverse map (token slot -> compact row) for quant dedup
if (write_inverse) {
ids_src1[it*n_expert_used + iex_used] = nex_prev + itc;
} else {
ids_src1[nex_prev + itc] = it*sis1 + iex_used % nchannels_y;
if (it_compact > 0) {
ggml_cuda_syncwarp();
dump_store(ids_src1, ids_dst, n_expert_used, nchannels_y, sis1, write_inverse, store, nex_prev, n_dumped, it_compact);
}
}
@@ -123,7 +220,7 @@ static __global__ void mm_ids_helper(
}
int col_low = nex_prev;
int col_high = nex_prev + min(int(tile_width.z), it_compact);
int col_high = nex_prev + min(int(tile_width.z), n_dumped);
// The while loop covers all tiles at the seams containing the column with % tile_width == 0.
while (col_high > fastdiv(col_low + tile_width.z - 1, tile_width) * tile_width.z) {
@@ -131,7 +228,7 @@ static __global__ void mm_ids_helper(
= {expert, col_low, col_high, /*padding =*/ 0};
col_low = col_high;
col_high = min(col_high + int(tile_width.z), nex_prev + it_compact);
col_high = min(col_high + int(tile_width.z), nex_prev + n_dumped);
}
// Finally, at most one extra tile_spec between the seams per expert.
@@ -157,10 +254,12 @@ static void launch_mm_ids_helper(
const dim3 num_blocks(n_experts, 1, 1);
const dim3 block_size(warp_size, 1, 1);
const size_t nbytes_shared = n_tokens*sizeof(mm_ids_helper_store);
GGML_ASSERT(nbytes_shared <= smpbo);
const size_t ne_shared_pad = n_expert_used_template == 0 ? std::max(n_expert_used_var, warp_size) : n_unroll*warp_size;
const size_t nbytes_store_core = std::min(smpbo - ne_shared_pad * sizeof(mm_ids_helper_store), n_tokens * sizeof(mm_ids_helper_store));
const size_t nbytes_shared = nbytes_store_core + ne_shared_pad * sizeof(mm_ids_helper_store);
const size_t ne_store = nbytes_shared / sizeof(mm_ids_helper_store);
mm_ids_helper<n_expert_used_template><<<num_blocks, block_size, nbytes_shared, stream>>>
(ids, ids_src1, ids_dst, tile_spec, n_tokens, n_expert_used_var, nchannels_y, tile_width_fd, si1, sis1, write_inverse);
(ids, ids_src1, ids_dst, tile_spec, n_tokens, n_expert_used_var, nchannels_y, tile_width_fd, si1, sis1, write_inverse, ne_store);
}
void ggml_cuda_launch_mm_ids_helper(
+50 -13
View File
@@ -92,13 +92,16 @@ static __global__ void rope_norm(const T * x,
} else if constexpr (std::is_same_v<half, D>) {
half2 v = make_half2(x0, x1);
ggml_cuda_memcpy_1<4>(dst + idst, &v);
} else if constexpr (std::is_same_v<nv_bfloat16, D>) {
const __align__(4) nv_bfloat16 v[2] = { ggml_cuda_cast<nv_bfloat16>(x0), ggml_cuda_cast<nv_bfloat16>(x1) };
ggml_cuda_memcpy_1<sizeof(v)>(dst + idst, v);
}
};
if (i0 < n_offs || i0 >= n_offs + n_dims) {
if (inplace) {
return;
}
store_coaelsced(x[ix + 0], x[ix + 1]);
store_coaelsced(ggml_cuda_cast<float>(x[ix + 0]), ggml_cuda_cast<float>(x[ix + 1]));
return;
}
@@ -113,8 +116,8 @@ static __global__ void rope_norm(const T * x,
rope_yarn<forward>(theta_base/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor, cos_theta, sin_theta);
const float x0 = x[ix + 0];
const float x1 = x[ix + 1];
const float x0 = ggml_cuda_cast<float>(x[ix + 0]);
const float x1 = ggml_cuda_cast<float>(x[ix + 1]);
store_coaelsced(x0 * cos_theta - x1 * sin_theta, x0 * sin_theta + x1 * cos_theta);
}
@@ -189,8 +192,8 @@ static __global__ void rope_neox(const T * x,
rope_yarn<forward>(theta_base/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor, cos_theta, sin_theta);
// idst/ix point at channel i0/2; the first channel of the rotated pair is n_offs + iw/2 = i0/2 + n_offs/2
const float x0 = x[ix + n_offs/2 + 0];
const float x1 = x[ix + n_offs/2 + n_dims/2];
const float x0 = ggml_cuda_cast<float>(x[ix + n_offs/2 + 0]);
const float x1 = ggml_cuda_cast<float>(x[ix + n_offs/2 + n_dims/2]);
dst[idst + n_offs/2 + 0] = ggml_cuda_cast<D>(x0 * cos_theta - x1 * sin_theta);
dst[idst + n_offs/2 + n_dims / 2] = ggml_cuda_cast<D>(x0 * sin_theta + x1 * cos_theta);
@@ -293,8 +296,8 @@ static __global__ void rope_multi(const T * x,
rope_yarn<forward>(theta_base/freq_factor, freq_scale, corr_dims, iw, ext_factor, attn_factor, cos_theta, sin_theta);
// idst/ix point at channel i0/2; the first channel of the rotated pair is n_offs + iw/2 = i0/2 + n_offs/2
const float x0 = x[ix + n_offs/2 + 0];
const float x1 = x[ix + n_offs/2 + n_dims/2];
const float x0 = ggml_cuda_cast<float>(x[ix + n_offs/2 + 0]);
const float x1 = ggml_cuda_cast<float>(x[ix + n_offs/2 + n_dims/2]);
dst[idst + n_offs/2 + 0] = ggml_cuda_cast<D>(x0*cos_theta - x1*sin_theta);
dst[idst + n_offs/2 + n_dims/2] = ggml_cuda_cast<D>(x0*sin_theta + x1*cos_theta);
@@ -357,11 +360,11 @@ static __global__ void rope_vision(const T * x,
rope_yarn<forward>(theta_base/freq_factor, freq_scale, corr_dims, i0, ext_factor, attn_factor, cos_theta, sin_theta);
const float x0 = x[ix + 0];
const float x1 = x[ix + n_dims];
const float x0 = ggml_cuda_cast<float>(x[ix + 0]);
const float x1 = ggml_cuda_cast<float>(x[ix + n_dims]);
dst[idst + 0] = x0*cos_theta - x1*sin_theta;
dst[idst + n_dims] = x0*sin_theta + x1*cos_theta;
dst[idst + 0] = ggml_cuda_cast<T>(x0*cos_theta - x1*sin_theta);
dst[idst + n_dims] = ggml_cuda_cast<T>(x0*sin_theta + x1*cos_theta);
}
template <bool forward, typename T, typename D>
@@ -569,8 +572,8 @@ void ggml_cuda_op_rope_impl(ggml_backend_cuda_context & ctx,
}
cudaStream_t stream = ctx.stream();
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16);
GGML_ASSERT( dst->type == GGML_TYPE_F32 || dst->type == GGML_TYPE_F16);
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16);
GGML_ASSERT( dst->type == GGML_TYPE_F32 || dst->type == GGML_TYPE_F16 || dst->type == GGML_TYPE_BF16);
// When not fused, src0 and dst types must match
// When fused (ROPE+VIEW+SET_ROWS), src0 may be F32 and dst may be F16
GGML_ASSERT(src0->type == dst->type || (src0->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F16));
@@ -655,6 +658,16 @@ void ggml_cuda_op_rope_impl(ggml_backend_cuda_context & ctx,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_BF16) {
rope_neox_cuda<forward, nv_bfloat16, nv_bfloat16>((const nv_bfloat16 *) src0_d, (nv_bfloat16 *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_F16) {
rope_neox_cuda<forward, nv_bfloat16, half>((const nv_bfloat16 *) src0_d, (half *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else {
GGML_ABORT("fatal error");
}
@@ -674,6 +687,16 @@ void ggml_cuda_op_rope_impl(ggml_backend_cuda_context & ctx,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, sections, is_imrope,
row_indices, set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_BF16) {
rope_multi_cuda<forward, nv_bfloat16, nv_bfloat16>((const nv_bfloat16 *) src0_d, (nv_bfloat16 *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, sections, is_imrope,
row_indices, set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_F16) {
rope_multi_cuda<forward, nv_bfloat16, half>((const nv_bfloat16 *) src0_d, (half *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, sections, is_imrope,
row_indices, set_rows_stride, inplace, stream);
} else {
GGML_ABORT("fatal error");
}
@@ -686,6 +709,10 @@ void ggml_cuda_op_rope_impl(ggml_backend_cuda_context & ctx,
rope_vision_cuda<forward>((const half *) src0_d, (half *) dst_d, ne00, ne01, ne02, s01, s02, s03, s1,
s2, s3, n_dims, nr, pos, freq_scale, freq_base, ext_factor, attn_factor,
corr_dims, freq_factors, sections, stream);
} else if (src0->type == GGML_TYPE_BF16) {
rope_vision_cuda<forward>((const nv_bfloat16 *) src0_d, (nv_bfloat16 *) dst_d, ne00, ne01, ne02, s01, s02, s03, s1,
s2, s3, n_dims, nr, pos, freq_scale, freq_base, ext_factor, attn_factor,
corr_dims, freq_factors, sections, stream);
} else {
GGML_ABORT("fatal error");
}
@@ -705,6 +732,16 @@ void ggml_cuda_op_rope_impl(ggml_backend_cuda_context & ctx,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_BF16) {
rope_norm_cuda<forward, nv_bfloat16, nv_bfloat16>((const nv_bfloat16 *) src0_d, (nv_bfloat16 *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else if (src0->type == GGML_TYPE_BF16 && dst_type == GGML_TYPE_F16) {
rope_norm_cuda<forward, nv_bfloat16, half>((const nv_bfloat16 *) src0_d, (half *) dst_d, ne00, ne01, ne02, s01, s02,
s03, s1, s2, s3, n_dims, n_offs, nr, pos, freq_scale, freq_base,
ext_factor, attn_factor, corr_dims, freq_factors, row_indices,
set_rows_stride, inplace, stream);
} else {
GGML_ABORT("fatal error");
}
+34 -19
View File
@@ -4,8 +4,8 @@
typedef void (*set_rows_kernel_t)(const char * src, char * dst);
// Generic quantized set_rows kernel template
template <typename idx_t, typename block_type, int qk, void (*quantize_func)(const float *, block_type *)>
static __global__ void k_set_rows_quant(const float * __restrict__ src0,
template <typename src_t, typename idx_t, typename block_type, int qk, void (*quantize_func)(const float *, block_type *)>
static __global__ void k_set_rows_quant(const src_t * __restrict__ src0,
const idx_t * __restrict__ src1,
block_type * __restrict__ dst,
const int64_t ne_total,
@@ -56,13 +56,23 @@ static __global__ void k_set_rows_quant(const float * __restrict__ src0,
ggml_cuda_pdl_sync();
const int64_t dst_row = *(src1 + i10*s10 + i11*s11 + i12*s12);
const float * src0_row = src0 + i01*s01 + i02*s02 + i03*s03;
const src_t * src0_row = src0 + i01*s01 + i02*s02 + i03*s03;
block_type * dst_row_ptr = dst + (dst_row*s1 + i02*s2 + i03*s3) / sizeof(block_type);
const float * src_block = src0_row + i00;
const src_t * src_block = src0_row + i00;
block_type * dst_block = dst_row_ptr + i00 / qk;
quantize_func(src_block, dst_block);
if constexpr (std::is_same_v<src_t, float>) {
quantize_func(src_block, dst_block);
} else {
// the quantize functions take F32 input: widen the block first
float src_block_f32[qk];
#pragma unroll
for (int j = 0; j < qk; ++j) {
src_block_f32[j] = ggml_cuda_cast<float>(src_block[j]);
}
quantize_func(src_block_f32, dst_block);
}
GGML_UNUSED(ne10);
GGML_UNUSED(ne11);
@@ -71,9 +81,9 @@ static __global__ void k_set_rows_quant(const float * __restrict__ src0,
}
// Template dispatch function for quantized set_rows
template<typename idx_t, typename block_type, int qk, void (*quantize_func)(const float*, block_type*)>
template<typename src_t, typename idx_t, typename block_type, int qk, void (*quantize_func)(const float*, block_type*)>
static void set_rows_cuda_quant(
const float * src0_d, const idx_t * src1_d, block_type * dst_d,
const src_t * src0_d, const idx_t * src1_d, block_type * dst_d,
const int64_t ne00, const int64_t ne01, const int64_t ne02, const int64_t ne03,
const int64_t ne10, const int64_t ne11, const int64_t ne12, const int64_t ne13,
const size_t nb01, const size_t nb02, const size_t nb03,
@@ -87,9 +97,9 @@ static void set_rows_cuda_quant(
const dim3 block_size(CUDA_SET_ROWS_BLOCK_SIZE);
const dim3 grid_size(num_blocks);
const int64_t s01 = nb01/sizeof(float);
const int64_t s02 = nb02/sizeof(float);
const int64_t s03 = nb03/sizeof(float);
const int64_t s01 = nb01/sizeof(src_t);
const int64_t s02 = nb02/sizeof(src_t);
const int64_t s03 = nb03/sizeof(src_t);
const int64_t s10 = nb10/sizeof(idx_t);
const int64_t s11 = nb11/sizeof(idx_t);
const int64_t s12 = nb12/sizeof(idx_t);
@@ -104,7 +114,7 @@ static void set_rows_cuda_quant(
const uint3 ne11_fd = init_fastdiv_values((uint32_t) ne11);
const uint3 ne12_fd = init_fastdiv_values((uint32_t) ne12);
k_set_rows_quant<idx_t, block_type, qk, quantize_func><<<grid_size, block_size, 0, stream>>>(
k_set_rows_quant<src_t, idx_t, block_type, qk, quantize_func><<<grid_size, block_size, 0, stream>>>(
src0_d, src1_d, dst_d, ne_total, ne10, ne11, ne12, ne13, s01, s02, s03, s10, s11, s12, s1, s2, s3, ne00_fd,
ne01_fd, ne02_fd, ne11_fd, ne12_fd);
}
@@ -268,7 +278,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_Q4_0) {
set_rows_cuda_quant<idx_t, block_q4_0, QK4_0, quantize_f32_q4_0_block>(
set_rows_cuda_quant<src_t, idx_t, block_q4_0, QK4_0, quantize_f32_q4_0_block>(
src0_d, src1_d, (block_q4_0*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -278,7 +288,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_Q4_1) {
set_rows_cuda_quant<idx_t, block_q4_1, QK4_1, quantize_f32_q4_1_block>(
set_rows_cuda_quant<src_t, idx_t, block_q4_1, QK4_1, quantize_f32_q4_1_block>(
src0_d, src1_d, (block_q4_1*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -288,7 +298,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_Q5_0) {
set_rows_cuda_quant<idx_t, block_q5_0, QK5_0, quantize_f32_q5_0_block>(
set_rows_cuda_quant<src_t, idx_t, block_q5_0, QK5_0, quantize_f32_q5_0_block>(
src0_d, src1_d, (block_q5_0*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -298,7 +308,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_Q5_1) {
set_rows_cuda_quant<idx_t, block_q5_1, QK5_1, quantize_f32_q5_1_block>(
set_rows_cuda_quant<src_t, idx_t, block_q5_1, QK5_1, quantize_f32_q5_1_block>(
src0_d, src1_d, (block_q5_1*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -308,7 +318,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_Q8_0) {
set_rows_cuda_quant<idx_t, block_q8_0, QK8_0, quantize_f32_q8_0_block>(
set_rows_cuda_quant<src_t, idx_t, block_q8_0, QK8_0, quantize_f32_q8_0_block>(
src0_d, src1_d, (block_q8_0*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -318,7 +328,7 @@ static void set_rows_cuda(ggml_backend_cuda_context & ctx, const ggml_tensor * s
stream
);
} else if (dst->type == GGML_TYPE_IQ4_NL) {
set_rows_cuda_quant<idx_t, block_iq4_nl, QK4_NL, quantize_f32_iq4_nl_block>(
set_rows_cuda_quant<src_t, idx_t, block_iq4_nl, QK4_NL, quantize_f32_iq4_nl_block>(
src0_d, src1_d, (block_iq4_nl*)dst->data,
ne00, ne01, ne02, ne03,
ne10, ne11, ne12, ne13,
@@ -382,12 +392,11 @@ void set_rows_cuda<half, int64_t>(ggml_backend_cuda_context & ctx, const ggml_te
}
}
void ggml_cuda_op_set_rows(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(src0->type == GGML_TYPE_F32 || (src0->type == GGML_TYPE_F16 && dst->type == GGML_TYPE_F16));
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16 || (src0->type == GGML_TYPE_F16 && dst->type == GGML_TYPE_F16));
GGML_ASSERT(src1->type == GGML_TYPE_I64 || src1->type == GGML_TYPE_I32);
if (src0->type == GGML_TYPE_F32) {
@@ -402,6 +411,12 @@ void ggml_cuda_op_set_rows(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
} else {
set_rows_cuda<half, int32_t>(ctx, src0, src1, dst);
}
} else if (src0->type == GGML_TYPE_BF16) {
if (src1->type == GGML_TYPE_I64) {
set_rows_cuda<nv_bfloat16, int64_t>(ctx, src0, src1, dst);
} else {
set_rows_cuda<nv_bfloat16, int32_t>(ctx, src0, src1, dst);
}
} else {
GGML_ABORT("unsupported type %s", ggml_type_name(src0->type));
}
+32
View File
@@ -0,0 +1,32 @@
#include "sleep.cuh"
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
// %globaltimer is a nanosecond wall clock, unlike clock64() it is unaffected by the SM clock and by frequency scaling
static __device__ __forceinline__ uint64_t globaltimer_ns() {
uint64_t t;
asm volatile("mov.u64 %0, %%globaltimer;" : "=l"(t));
return t;
}
// a single thread is enough, the next kernel on the same stream cannot start before this one retires
static __global__ void sleep_ns(const uint64_t ns) {
const uint64_t t0 = globaltimer_ns();
while (globaltimer_ns() - t0 < ns) {}
}
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
sleep_ns<<<1, 1, 0, ctx.stream()>>>(1000*(uint64_t) ggml_get_op_params_i32(dst, 0));
CUDA_CHECK(cudaGetLastError());
}
#else
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
GGML_UNUSED(ctx);
GGML_UNUSED(dst);
GGML_ABORT("GGML_OP_SLEEP requires the %%globaltimer register, which is only available on CUDA");
}
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
+3
View File
@@ -0,0 +1,3 @@
#include "common.cuh"
void ggml_cuda_op_sleep(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
+2 -9
View File
@@ -184,13 +184,6 @@ static __global__ void top_k_radix_gather(
constexpr int NBINS = 1 << RADIX_BITS;
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
constexpr int nwarps = BLOCK_SIZE / warp_size;
const auto popc = [](const ggml_cuda_thread_mask_t mask) {
if constexpr (warp_size == 64) {
return __popcll(mask);
} else {
return __popc(mask);
}
};
__shared__ int warp_ties[nwarps];
const int row = blockIdx.x / blocks_per_row;
@@ -247,11 +240,11 @@ static __global__ void top_k_radix_gather(
const bool is_tie = col < warp_end && top_k_float_to_ordered(row_src[col]) == threshold;
const ggml_cuda_thread_mask_t ties = ggml_cuda_ballot(is_tie);
const int tie_pos = pos + popc(ties & (((ggml_cuda_thread_mask_t) 1 << lane) - 1));
const int tie_pos = pos + ggml_cuda_popc(ties & (((ggml_cuda_thread_mask_t) 1 << lane) - 1));
if (is_tie && tie_pos < pos_end) {
row_dst[k - rank + tie_pos] = col;
}
pos += popc(ties);
pos += ggml_cuda_popc(ties);
}
}
+2 -2
View File
@@ -1713,7 +1713,7 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
return has_simdgroup_reduction && op->src[0]->type == GGML_TYPE_F32 && (ggml_is_contiguous_rows(op->src[0]));
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK:
return ggml_rope_get_truncate(op);
return ggml_rope_get_truncate(op) && (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16);
case GGML_OP_IM2COL:
return ggml_is_contiguous(op->src[1]) && op->src[1]->type == GGML_TYPE_F32 && (op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_F32);
case GGML_OP_CONV_2D:
@@ -2375,7 +2375,7 @@ void ggml_metal_buffer_memset_tensor(ggml_metal_buffer_t buf, struct ggml_tensor
id<MTLBlitCommandEncoder> encoder = [cmd_buf blitCommandEncoder];
[encoder fillBuffer:bid_dst.metal
range:NSMakeRange(bid_dst.offs, bid_dst.offs + size)
range:NSMakeRange(bid_dst.offs, size)
value:value];
[encoder endEncoding];
+3
View File
@@ -323,6 +323,9 @@ set(GGML_OPENCL_KERNELS
mul_mm_f16_f32_kq_kqv
mul_mm_f16_f32_kq_p8
mul_mm_q8_f32_kqv
mul_mm_q8_kq_p8
mul_mm_q8_kq_p8_dk512
flash_attn_q8_rows
conv2d
conv2d_f16_f32
conv2_dw
+235 -55
View File
@@ -781,6 +781,10 @@ struct ggml_backend_opencl_context {
ggml_cl_buffer prealloc_fa_pd;
ggml_cl_buffer prealloc_fa_bmax;
ggml_cl_buffer prealloc_fa_bsum;
ggml_cl_buffer prealloc_fa_k8;
ggml_cl_buffer prealloc_fa_k8_scale;
ggml_cl_buffer prealloc_fa_q8;
ggml_cl_buffer prealloc_fa_q8_scale;
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
ggml_cl_buffer prealloc_adreno_xmem_const;
@@ -899,6 +903,10 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_fa_kqv_direct_f16 = nullptr;
cl_kernel kernel_fa_kqv_direct_reduce = nullptr;
cl_kernel kernel_fa_v_transpose_q8 = nullptr;
cl_kernel kernel_fa_q8_rows_f16 = nullptr;
cl_kernel kernel_fa_q8_rows_f32 = nullptr;
cl_kernel kernel_mul_mm_q8_kq_p8 = nullptr;
cl_kernel kernel_mul_mm_q8_kq_p8_dk512 = nullptr;
ggml_opencl_fa_kernels fa;
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
ggml_cl_adreno_xmem_attn_state adreno_xmem_attn;
@@ -1506,7 +1514,18 @@ struct ggml_backend_opencl_context {
#endif // GGML_OPENCL_USE_ADRENO_KERNELS
~ggml_backend_opencl_context() {
for (cl_kernel kernel : {kernel_mul_mm_f16_f32_kq_p8, kernel_mul_mm_q8_f32_kqv, kernel_fa_p8_fixup, kernel_fa_v_transpose_q8, kernel_fa_kqv_direct_f16, kernel_fa_kqv_direct_reduce}) {
for (cl_kernel kernel : {
kernel_mul_mm_f16_f32_kq_p8,
kernel_mul_mm_q8_f32_kqv,
kernel_fa_p8_fixup,
kernel_fa_v_transpose_q8,
kernel_fa_kqv_direct_f16,
kernel_fa_kqv_direct_reduce,
kernel_fa_q8_rows_f16,
kernel_fa_q8_rows_f32,
kernel_mul_mm_q8_kq_p8,
kernel_mul_mm_q8_kq_p8_dk512
}) {
if (kernel != nullptr) {
CL_CHECK(clReleaseKernel(kernel));
}
@@ -1648,7 +1667,7 @@ static cl_program build_program_from_source_ex(cl_context ctx, cl_device_id dev,
return NULL;
}
static cl_program build_program_from_source(ggml_backend_opencl_context * backend_ctx, const char* program_buffer, const std::string &compile_opts) {
static cl_program build_program_from_source(ggml_backend_opencl_context * backend_ctx, const char* program_buffer, const std::string &compile_opts, bool fatal = true) {
cl_context ctx = backend_ctx->context;
cl_device_id dev = backend_ctx->device;
@@ -1660,7 +1679,7 @@ static cl_program build_program_from_source(ggml_backend_opencl_context * backen
return p_cached;
}
cl_program p = build_program_from_source_ex(ctx, dev, program_buffer, compile_opts, /*fatal=*/true);
cl_program p = build_program_from_source_ex(ctx, dev, program_buffer, compile_opts, fatal);
// Best-effort save of the freshly-built binary (no-op if cache disabled).
if (p != nullptr) {
@@ -3995,6 +4014,52 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
}
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (backend_ctx->gpu_family == ADRENO && backend_ctx->has_integer_dot) {
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string rows_src {
#include "flash_attn_q8_rows.cl.h"
};
const std::string kq_src {
#include "mul_mm_q8_kq_p8.cl.h"
};
#else
const std::string rows_src = read_file("flash_attn_q8_rows.cl");
const std::string kq_src = read_file("mul_mm_q8_kq_p8.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, rows_src.c_str(), compile_opts, /*fatal=*/false);
if (prog != nullptr) {
CL_CHECK((backend_ctx->kernel_fa_q8_rows_f16 = clCreateKernel(prog, "kernel_fa_q8_rows_f16", &err), err));
CL_CHECK((backend_ctx->kernel_fa_q8_rows_f32 = clCreateKernel(prog, "kernel_fa_q8_rows_f32", &err), err));
CL_CHECK(clReleaseProgram(prog));
}
prog = build_program_from_source(backend_ctx, kq_src.c_str(), compile_opts, /*fatal=*/false);
if (prog != nullptr) {
CL_CHECK((backend_ctx->kernel_mul_mm_q8_kq_p8 = clCreateKernel(prog, "kernel_mul_mm_q8_kq_p8", &err), err));
CL_CHECK(clReleaseProgram(prog));
}
GGML_LOG_CONT(".");
}
// mul_mm_q8_kq_p8_dk512
if (backend_ctx->gpu_family == ADRENO && backend_ctx->has_integer_dot) {
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "mul_mm_q8_kq_p8_dk512.cl.h"
};
#else
const std::string kernel_src = read_file("mul_mm_q8_kq_p8_dk512.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), compile_opts, /*fatal=*/false);
if (prog != nullptr) {
cl_kernel kernel = clCreateKernel(prog, "kernel_mul_mm_q8_kq_p8_dk512", &err);
CL_CHECK(clReleaseProgram(prog));
if (err == CL_SUCCESS) {
backend_ctx->kernel_mul_mm_q8_kq_p8_dk512 = kernel;
}
}
GGML_LOG_CONT(".");
}
// mul_mm_f16_f32_kq_p8
if (backend_ctx->gpu_family == ADRENO && backend_ctx->has_integer_dot) {
#ifdef GGML_OPENCL_EMBED_KERNELS
@@ -12305,7 +12370,7 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
}
return false;
}
return true;
return op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16;
}
case GGML_OP_SOLVE_TRI:
return op->src[0]->type == GGML_TYPE_F32 && ggml_is_contiguous(op->src[0]);
@@ -22922,12 +22987,58 @@ static void ggml_cl_flash_attn_prefill_bin(ggml_backend_t backend, const ggml_te
}
#endif // GGML_OPENCL_USE_ADRENO_KERNELS
static void ggml_cl_fa_q8_rows(
ggml_backend_t backend, const ggml_tensor * src, const ggml_tensor * dst,
cl_mem values, cl_mem scales, int64_t row0, int nrow
) {
auto * backend_ctx = (ggml_backend_opencl_context *) backend->context;
auto * extra = (ggml_tensor_extra_cl *) (src->view_src ? src->view_src->extra : src->extra);
const cl_ulong offset = extra->offset + src->view_offs + row0 * src->nb[1];
const cl_ulong nb1 = src->nb[1];
const cl_ulong nb2 = src->nb[2];
const int dk = src->ne[0];
const int nhead = src->ne[2];
cl_kernel kernel;
switch (src->type) {
case GGML_TYPE_F16: kernel = backend_ctx->kernel_fa_q8_rows_f16; break;
case GGML_TYPE_F32: kernel = backend_ctx->kernel_fa_q8_rows_f32; break;
default: GGML_ABORT("unsupported int8 KQ input type");
}
cl_uint a = 0;
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &extra->data_device));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &offset));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &nb1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &nb2));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &values));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &scales));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &dk));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &nrow));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &nhead));
const size_t blocks = (size_t) (dk / 32) * nrow * nhead;
size_t gws[3] = {(blocks + 63) / 64 * 64, 1, 1};
size_t lws[3] = {64, 1, 1};
backend_ctx->enqueue_ndrange_kernel(kernel, 3, gws, lws, dst);
}
static void ggml_cl_fa_kq_int8_prepare(ggml_backend_t backend, const ggml_tensor * dst, int64_t n_q_chunk) {
auto * backend_ctx = (ggml_backend_opencl_context *) backend->context;
const ggml_tensor * q = dst->src[0];
const ggml_tensor * k = dst->src[1];
const size_t k_bytes = (size_t) k->ne[0] * k->ne[1] * k->ne[2];
const size_t q_bytes = (size_t) q->ne[0] * n_q_chunk * q->ne[2];
backend_ctx->prealloc_fa_k8.allocate(backend_ctx->context, k_bytes);
backend_ctx->prealloc_fa_k8_scale.allocate(backend_ctx->context, k_bytes / 32 * sizeof(cl_half));
backend_ctx->prealloc_fa_q8.allocate(backend_ctx->context, q_bytes);
backend_ctx->prealloc_fa_q8_scale.allocate(backend_ctx->context, q_bytes / 32 * sizeof(cl_half));
ggml_cl_fa_q8_rows(backend, k, dst, backend_ctx->prealloc_fa_k8.buffer, backend_ctx->prealloc_fa_k8_scale.buffer, 0, k->ne[1]);
}
static void ggml_cl_fa_kq_p8_dispatch(
ggml_backend_t backend,
const ggml_tensor * q, const ggml_tensor * k,
const ggml_tensor * mask, const ggml_tensor * sinks,
const ggml_tensor * dst,
int64_t q0, int nqc, float scale, float max_bias, int p_pitch
int64_t q0, int nqc, float scale, float max_bias, int p_pitch, bool kq_int8
) {
auto * backend_ctx = (ggml_backend_opencl_context *) backend->context;
auto * extra_k = (ggml_tensor_extra_cl *) (k->view_src ? k->view_src->extra : k->extra);
@@ -22940,64 +23051,94 @@ static void ggml_cl_fa_kq_p8_dispatch(
const int D_A = k->ne[2];
const int D_B = q->ne[2];
cl_int err;
cl_buffer_region region = {extra_k->offset + k->view_offs, k->nb[1] * k->ne[1]};
cl_mem k_sub = clCreateSubBuffer(extra_k->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err);
CL_CHECK(err);
cl_image_format fmt = {CL_RGBA, CL_FLOAT};
cl_image_desc desc = {};
desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
desc.image_width = region.size / 16;
desc.buffer = k_sub;
cl_mem k_img = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &fmt, &desc, nullptr, &err);
CL_CHECK(err);
region.origin = extra_q->offset + q->view_offs + q0 * q->nb[1];
region.size = (size_t) K * N * D_B * sizeof(float);
cl_mem q_sub = clCreateSubBuffer(extra_q->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err);
CL_CHECK(err);
const cl_ulong off_mask = extra_m->offset + mask->view_offs + q0 * mask->nb[1];
const cl_ulong mask_nb1 = mask->nb[1];
const int n_head_log2 = 1u << (unsigned) floorf(log2f((float) D_B));
const float m0 = powf(2.0f, -max_bias / n_head_log2);
const float m1 = powf(2.0f, -(max_bias / 2.0f) / n_head_log2);
cl_kernel kernel;
cl_uint a = 0;
cl_kernel kernel = backend_ctx->kernel_mul_mm_f16_f32_kq_p8;
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &k_img));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &q_sub));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_pq.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bmax.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bsum.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &extra_m->data_device));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &off_mask));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &mask_nb1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &scale));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &max_bias));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m0));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &n_head_log2));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &M));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &K));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &N));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_A));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_B));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &p_pitch));
if (kq_int8) {
ggml_cl_fa_q8_rows(backend, q, dst, backend_ctx->prealloc_fa_q8.buffer, backend_ctx->prealloc_fa_q8_scale.buffer, q0, nqc);
kernel = K <= 256 ? backend_ctx->kernel_mul_mm_q8_kq_p8 : backend_ctx->kernel_mul_mm_q8_kq_p8_dk512;
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_k8.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_k8_scale.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_q8.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_q8_scale.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_pq.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bmax.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bsum.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &extra_m->data_device));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &off_mask));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &mask_nb1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &scale));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &max_bias));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m0));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &n_head_log2));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &K));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &M));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &N));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_B));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_A));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &p_pitch));
size_t gws[3] = {64 * (((size_t) N + 31) / 32) * (D_B / D_A), ((size_t) M + 511) / 512, (size_t) D_A};
size_t lws[3] = {64, 1, 1};
backend_ctx->enqueue_ndrange_kernel(kernel, 3, gws, lws, dst);
} else {
cl_int err;
const size_t n_tiles = (N + 31) / 32;
size_t gws[3] = {64, n_tiles, (size_t)(M / 64) * D_B};
cl_buffer_region region = {extra_k->offset + k->view_offs, k->nb[1] * k->ne[1]};
cl_mem k_sub = clCreateSubBuffer(extra_k->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err);
CL_CHECK(err);
const bool two_tiles = n_tiles % 2 == 0 && backend_ctx->get_kernel_workgroup_size(kernel) >= 128;
size_t lws[3] = {64, two_tiles ? (size_t) 2 : (size_t) 1, 1};
cl_image_format fmt = {CL_RGBA, CL_FLOAT};
cl_image_desc desc = {};
desc.image_type = CL_MEM_OBJECT_IMAGE1D_BUFFER;
desc.image_width = region.size / 16;
desc.buffer = k_sub;
cl_mem k_img = clCreateImage(backend_ctx->context, CL_MEM_READ_ONLY, &fmt, &desc, nullptr, &err);
CL_CHECK(err);
backend_ctx->enqueue_ndrange_kernel(kernel, 3, gws, lws, dst);
CL_CHECK(clReleaseMemObject(k_img));
CL_CHECK(clReleaseMemObject(k_sub));
CL_CHECK(clReleaseMemObject(q_sub));
region.origin = extra_q->offset + q->view_offs + q0 * q->nb[1];
region.size = (size_t) K * N * D_B * sizeof(float);
cl_mem q_sub = clCreateSubBuffer(extra_q->data_device, 0, CL_BUFFER_CREATE_TYPE_REGION, &region, &err);
CL_CHECK(err);
kernel = backend_ctx->kernel_mul_mm_f16_f32_kq_p8;
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &k_img));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &q_sub));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_pq.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bmax.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &backend_ctx->prealloc_fa_bsum.buffer));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_mem), &extra_m->data_device));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &off_mask));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(cl_ulong), &mask_nb1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &scale));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &max_bias));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m0));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(float), &m1));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &n_head_log2));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &M));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &K));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &N));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_A));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &D_B));
CL_CHECK(clSetKernelArg(kernel, a++, sizeof(int), &p_pitch));
const size_t n_tiles = (N + 31) / 32;
size_t gws[3] = {64, n_tiles, (size_t)(M / 64) * D_B};
const bool two_tiles = n_tiles % 2 == 0 && backend_ctx->get_kernel_workgroup_size(kernel) >= 128;
size_t lws[3] = {64, two_tiles ? (size_t) 2 : (size_t) 1, 1};
backend_ctx->enqueue_ndrange_kernel(kernel, 3, gws, lws, dst);
CL_CHECK(clReleaseMemObject(k_img));
CL_CHECK(clReleaseMemObject(k_sub));
CL_CHECK(clReleaseMemObject(q_sub));
}
cl_mem sinks_buf = extra_q->data_device;
cl_ulong sinks_off = 0;
@@ -23261,7 +23402,7 @@ static void ggml_cl_flash_attn_decompose_kq_p8(
ggml_backend_t backend,
const ggml_tensor * q, const ggml_tensor * k,
ggml_tensor * dst,
int64_t n_q_chunk, float scale, float max_bias, bool kqv_direct
int64_t n_q_chunk, float scale, float max_bias, bool kqv_direct, bool kq_int8
) {
auto * backend_ctx = (ggml_backend_opencl_context *) backend->context;
const ggml_tensor * v = dst->src[2];
@@ -23345,13 +23486,17 @@ static void ggml_cl_flash_attn_decompose_kq_p8(
backend_ctx->enqueue_ndrange_kernel(kernel, 3, gws, lws, dst);
}
if (kq_int8) {
ggml_cl_fa_kq_int8_prepare(backend, dst, n_q_chunk);
}
ggml_tensor_extra_cl extra_kqv;
ggml_tensor kqv;
for (int64_t q0 = 0; q0 < n_q; q0 += n_q_chunk) {
const int64_t nqc = MIN(n_q_chunk, n_q - q0);
ggml_cl_fa_kq_p8_dispatch(backend, q, k, mask, sinks, dst, q0, nqc, scale, max_bias, kv_pitch);
ggml_cl_fa_kq_p8_dispatch(backend, q, k, mask, sinks, dst, q0, nqc, scale, max_bias, kv_pitch, kq_int8);
ggml_cl_fa_decompose_scratch_tensor(kqv, extra_kqv, backend_ctx->prealloc_fa_kqv.buffer, GGML_TYPE_F32, dv, nqc, n_head);
@@ -23408,6 +23553,7 @@ static void ggml_cl_flash_attn_decompose_kq_p8(
// head size. GGML_OPENCL_FA_DECOMPOSE_MB sets the KQ scratch budget (default 256).
// GGML_OPENCL_FA_KQ_P8=0 disables U8 probabilities and int8 KQV.
// GGML_OPENCL_FA_KQV_DIRECT=0 disables direct PV for small query batches.
// GGML_OPENCL_FA_KQ_INT8=0 keeps F16 KQ; =1 enables int8 KQ for supported shapes.
static bool ggml_cl_flash_attn_decompose_dispatch(
ggml_backend_t backend,
const ggml_tensor * q, const ggml_tensor * k, ggml_tensor * dst
@@ -23537,12 +23683,46 @@ static bool ggml_cl_flash_attn_decompose_dispatch(
return false;
}
static const int kq_int8_env = [] {
const char * e = getenv("GGML_OPENCL_FA_KQ_INT8");
return (e && e[0]) ? (e[0] != '0' ? 1 : 0) : -1;
}();
static const int dk512_env = [] {
const char * e = getenv("GGML_OPENCL_FA_KQ_DK512_F16");
return (e && e[0]) ? (e[0] != '0' ? 1 : 0) : -1;
}();
const bool dk512 = dk > 256 && dk <= 512 && n_q >= 32 &&
(dk512_env == 1 || (dk512_env < 0 && backend_ctx->adreno_gen == ADRENO_GPU_GEN::X2E));
const bool kq_int8_possible = kq_p8 && (dk <= 256 || dk512) && dk % 32 == 0 &&
backend_ctx->kernel_fa_q8_rows_f16 != nullptr && backend_ctx->kernel_fa_q8_rows_f32 != nullptr &&
n_head <= INT_MAX && n_head_kv <= INT_MAX &&
n_q <= INT_MAX / n_head / (dk / 32) && n_kv <= INT_MAX / n_head_kv / (dk / 32) &&
(dst->src[4] == nullptr || dk > 64) &&
(kq_int8_env == 1 || (kq_int8_env < 0 && backend_ctx->adreno_gen == ADRENO_GPU_GEN::X2E && n_kv >= 2048)) &&
backend_ctx->get_kernel_workgroup_size(backend_ctx->kernel_fa_q8_rows_f16) >= 64 &&
backend_ctx->get_kernel_workgroup_size(backend_ctx->kernel_fa_q8_rows_f32) >= 64;
cl_kernel kq_int8_kernel = nullptr;
if (kq_int8_possible) {
kq_int8_kernel = dk <= 256 ? backend_ctx->kernel_mul_mm_q8_kq_p8 : backend_ctx->kernel_mul_mm_q8_kq_p8_dk512;
}
const bool kq_int8 = kq_int8_kernel != nullptr && backend_ctx->get_kernel_workgroup_size(kq_int8_kernel) >= 64;
// Fewer query tiles per KV block reduce cache pressure in the int8 KQ kernel.
static const int64_t kq_int8_max_chunk = [] {
const char * e = getenv("GGML_OPENCL_FA_KQ_INT8_MAX_CHUNK");
const int v = (e && e[0]) ? atoi(e) : 96;
return (int64_t) (v >= 32 ? (v / 32) * 32 : 96);
}();
if (kq_int8) {
n_q_chunk = MIN(n_q_chunk, kq_int8_max_chunk);
}
// even out the chunks, then align to the 32-wide n-tile
const int64_t n_chunks = (n_q + n_q_chunk - 1) / n_q_chunk;
n_q_chunk = MIN(n_q_chunk, ((n_q + n_chunks - 1) / n_chunks + 31) & ~(int64_t) 31);
if (kq_p8) {
ggml_cl_flash_attn_decompose_kq_p8(backend, q, k, dst, n_q_chunk, scale, max_bias, kqv_direct);
ggml_cl_flash_attn_decompose_kq_p8(backend, q, k, dst, n_q_chunk, scale, max_bias, kqv_direct, kq_int8);
} else {
ggml_cl_flash_attn_decompose(backend, q, k, dst, n_q_chunk, scale, max_bias);
}
@@ -0,0 +1,77 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
// Store 32-element blocks as separate int8 values and half scales.
kernel void kernel_fa_q8_rows_f16(
global const char * src, ulong off_src, ulong nb1, ulong nb2,
global char * dq, global half * dd,
int ne0, int nrow, int nhead
) {
const int nblk = ne0 / 32;
const int gid = get_global_id(0);
if (gid >= nblk*nrow*nhead) {
return;
}
const int b = gid % nblk;
const int r = (gid / nblk) % nrow;
const int h = gid / (nblk*nrow);
global const half * p = (global const half *)(src + off_src + (ulong)h*nb2 + (ulong)r*nb1) + b*32;
float v[32];
float amax = 0.0f;
#pragma unroll
for (int i = 0; i < 8; ++i) {
const float4 x = convert_float4(vload4(i, p));
v[i*4+0] = x.s0;
v[i*4+1] = x.s1;
v[i*4+2] = x.s2;
v[i*4+3] = x.s3;
amax = fmax(amax, fmax(fmax(fabs(x.s0), fabs(x.s1)), fmax(fabs(x.s2), fabs(x.s3))));
}
const float d = amax / 127.0f;
const float id = amax > 0.0f ? 127.0f / amax : 0.0f;
const size_t row = (size_t)h*nrow + r;
dd[row*nblk + b] = (half)d;
global char * q = dq + row*ne0 + b*32;
#pragma unroll
for (int i = 0; i < 32; ++i) {
q[i] = (char)clamp((int)rint(v[i]*id), -127, 127);
}
}
kernel void kernel_fa_q8_rows_f32(
global const char * src, ulong off_src, ulong nb1, ulong nb2,
global char * dq, global half * dd,
int ne0, int nrow, int nhead
) {
const int nblk = ne0 / 32;
const int gid = get_global_id(0);
if (gid >= nblk*nrow*nhead) {
return;
}
const int b = gid % nblk;
const int r = (gid / nblk) % nrow;
const int h = gid / (nblk*nrow);
global const float * p = (global const float *)(src + off_src + (ulong)h*nb2 + (ulong)r*nb1) + b*32;
float v[32];
float amax = 0.0f;
#pragma unroll
for (int i = 0; i < 8; ++i) {
const float4 x = vload4(i, p);
v[i*4+0] = x.s0;
v[i*4+1] = x.s1;
v[i*4+2] = x.s2;
v[i*4+3] = x.s3;
amax = fmax(amax, fmax(fmax(fabs(x.s0), fabs(x.s1)), fmax(fabs(x.s2), fabs(x.s3))));
}
const float d = amax / 127.0f;
const float id = amax > 0.0f ? 127.0f / amax : 0.0f;
const size_t row = (size_t)h*nrow + r;
dd[row*nblk + b] = (half)d;
global char * q = dq + row*ne0 + b*32;
#pragma unroll
for (int i = 0; i < 32; ++i) {
q[i] = (char)clamp((int)rint(v[i]*id), -127, 127);
}
}
@@ -0,0 +1,174 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_khr_subgroups : enable
#ifdef cl_khr_integer_dot_product
#pragma OPENCL EXTENSION cl_khr_integer_dot_product : enable
#endif
#define KQ_TN 32
#define KQ_MB 8
#define KQ_TM 64
#define KQ_WG 64
#define KQ_DK_MAX 256
__attribute__((qcom_wave_pair_mode(1)))
kernel void kernel_mul_mm_q8_kq_p8(
global const uint * kq, // K int8, [head_kv][n_kv][dk] as packed uints
global const half * kd, // K scales, [head_kv][n_kv][dk/32]
global const uint * qq, // Q int8, [head][n_q][dk] as packed uints
global const half * qd, // Q scales, [head][n_q][dk/32]
global uchar * qp, // u8 P, [head][n_q][p_pitch]
global float * bmax, // per 32-row block max, [head][n_q][p_pitch/32]
global float * bsum, // per block sum of exp(s - bmax), [head][n_q][p_pitch/32]
global const char * mask, // f16 [>=n_kv][n_q] view, row stride mask_nb1 bytes
ulong offset_mask,
ulong mask_nb1,
float scale,
float max_bias,
float m0,
float m1,
int n_head_log2,
int dk,
int n_kv,
int n_q,
int n_head,
int n_head_kv,
int p_pitch // bytes per P row, >= n_kv, multiple of 32
) {
const int lid = get_local_id(0);
const int gsz = n_head / n_head_kv;
const int g0 = get_group_id(0);
const int head_kv = get_group_id(2);
const int head = head_kv*gsz + (g0 % gsz);
const int qn0 = (g0 / gsz) * KQ_TN;
const int kv0 = get_group_id(1)*KQ_TM*KQ_MB + lid;
const int nblk = dk / 32;
const int nu = dk / 4;
__local uint sh_q [KQ_TN][KQ_DK_MAX/4];
__local half sh_qd[KQ_TN][KQ_DK_MAX/32];
__local float p8_lds[1024];
const size_t qbase = (size_t)head*n_q;
for (int i = lid; i < KQ_TN*(KQ_DK_MAX/16); i += KQ_WG) {
const int t = i / (KQ_DK_MAX/16);
const int u = (i % (KQ_DK_MAX/16)) * 4;
const int qi = qn0 + t;
uint4 v = (uint4)(0u);
if (qi < n_q && u < nu) {
v = vload4(0, &qq[(qbase + qi)*nu + u]);
}
vstore4(v, 0, &sh_q[t][u]);
}
for (int i = lid; i < KQ_TN*(KQ_DK_MAX/32); i += KQ_WG) {
const int t = i / (KQ_DK_MAX/32);
const int b = i % (KQ_DK_MAX/32);
const int qi = qn0 + t;
sh_qd[t][b] = (qi < n_q && b < nblk) ? qd[(qbase + qi)*nblk + b] : (half)0.0f;
}
barrier(CLK_LOCAL_MEM_FENCE);
float slope = 1.0f;
if (max_bias > 0.0f) {
const float base = head < n_head_log2 ? m0 : m1;
const int ex = head < n_head_log2 ? head + 1 : 2*(head - n_head_log2) + 1;
slope = pow(base, ex);
}
global const half * mrow = (global const half *)(mask + offset_mask);
const uint mstride = (uint)(mask_nb1 / 2);
const int pblk = p_pitch / 32;
for (int m = 0; m < KQ_MB; ++m) {
const int kv = kv0 + m*KQ_TM;
if (kv - lid >= n_kv) {
break; // uniform across the workgroup
}
float acc[KQ_TN];
#pragma unroll
for (int t = 0; t < KQ_TN; ++t) {
acc[t] = 0.0f;
}
const int kl = min(kv, n_kv - 1);
const size_t kbase = ((size_t)head_kv*n_kv + kl)*nu;
const size_t kdbas = ((size_t)head_kv*n_kv + kl)*nblk;
for (int b = 0; b < nblk; ++b) {
const uint4 w0 = vload4(0, &kq[kbase + b*8]);
const uint4 w1 = vload4(0, &kq[kbase + b*8 + 4]);
const float dks = (float)kd[kdbas + b];
#pragma unroll
for (int t = 0; t < KQ_TN; ++t) {
const uint4 a0 = vload4(0, &sh_q[t][b*8]);
const uint4 a1 = vload4(0, &sh_q[t][b*8 + 4]);
int raw = 0;
raw = dot_acc_sat_4x8packed_ss_int(w0.s0, a0.s0, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s1, a0.s1, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s2, a0.s2, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s3, a0.s3, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s0, a1.s0, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s1, a1.s1, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s2, a1.s2, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s3, a1.s3, raw);
acc[t] += dks * (float)sh_qd[t][b] * (float)raw;
}
}
// Each lane owns one complete 32-row softmax block after the transpose.
const int kvblk0 = kv - lid;
#pragma unroll
for (int half_ = 0; half_ < 2; ++half_) {
barrier(CLK_LOCAL_MEM_FENCE);
#pragma unroll
for (int j = 0; j < 16; ++j) {
p8_lds[j*64 + lid] = acc[half_*16 + j];
}
barrier(CLK_LOCAL_MEM_FENCE);
if (lid < 32) {
const int j = lid >> 1;
const int b = lid & 1;
const int n = qn0 + half_*16 + j;
if (n < n_q) {
const int kvb = kvblk0 + b*32;
const __local float * src = p8_lds + j*64 + b*32;
float16 s0 = vload16(0, src);
float16 s1 = vload16(1, src);
const global half * mp = mrow + (size_t)n*mstride + kvb;
const float16 m0v = convert_float16(vload16(0, mp));
const float16 m1v = convert_float16(vload16(1, mp));
s0 = s0*scale + slope*m0v;
s1 = s1*scale + slope*m1v;
const float16 mx16 = fmax(s0, s1);
const float8 mx8 = fmax(mx16.lo, mx16.hi);
const float4 mx4 = fmax(mx8.lo, mx8.hi);
const float2 mx2 = fmax(mx4.lo, mx4.hi);
const float amax = fmax(mx2.x, mx2.y);
const size_t prow = qbase + n;
const int blk = kvb >> 5;
float sum = 0.0f;
uchar16 q0 = (uchar16)(0);
uchar16 q1 = (uchar16)(0);
// finite sentinel: the program builds with -cl-finite-math-only
if (amax > -1.0e30f) {
const float16 e0 = exp(s0 - amax);
const float16 e1 = exp(s1 - amax);
const float16 a16 = e0 + e1;
const float8 a8 = a16.lo + a16.hi;
const float4 a4 = a8.lo + a8.hi;
const float2 a2 = a4.lo + a4.hi;
sum = a2.x + a2.y;
q0 = convert_uchar16_sat_rte(e0*255.0f);
q1 = convert_uchar16_sat_rte(e1*255.0f);
}
global uchar * qdst = qp + prow*(size_t)p_pitch + (size_t)kvb;
vstore16(q0, 0, qdst);
vstore16(q1, 1, qdst);
bmax[prow*pblk + blk] = amax;
bsum[prow*pblk + blk] = sum;
}
}
}
}
}
@@ -0,0 +1,170 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_khr_subgroups : enable
#ifdef cl_khr_integer_dot_product
#pragma OPENCL EXTENSION cl_khr_integer_dot_product : enable
#endif
#define KQ_TN 32
#define KQ_MB 8
#define KQ_TM 64
#define KQ_WG 64
#define KQ_DKC 256
__attribute__((qcom_wave_pair_mode(1)))
kernel void kernel_mul_mm_q8_kq_p8_dk512(
global const uint * kq, // K int8, [head_kv][n_kv][dk] as packed uints
global const half * kd, // K scales, [head_kv][n_kv][dk/32]
global const uint * qq, // Q int8, [head][n_q][dk] as packed uints
global const half * qd, // Q scales, [head][n_q][dk/32]
global uchar * qp, // u8 P, [head][n_q][p_pitch]
global float * bmax, // per 32-row block max, [head][n_q][p_pitch/32]
global float * bsum, // per block sum of exp(s - bmax), [head][n_q][p_pitch/32]
global const char * mask, // f16 [>=n_kv][n_q] view, row stride mask_nb1 bytes
ulong offset_mask,
ulong mask_nb1,
float scale,
float max_bias,
float m0,
float m1,
int n_head_log2,
int dk,
int n_kv,
int n_q,
int n_head,
int n_head_kv,
int p_pitch // bytes per P row, >= n_kv, multiple of 32
) {
const int lid = get_local_id(0);
const int gsz = n_head / n_head_kv;
const int g0 = get_group_id(0);
const int head_kv = get_group_id(2);
const int head = head_kv*gsz + (g0 % gsz);
const int qn0 = (g0 / gsz) * KQ_TN;
const int kv0 = get_group_id(1)*KQ_TM*KQ_MB + lid;
const int nblk = dk / 32;
const int nu = dk / 4;
// Reuse the 256-wide Q slice for the F32 softmax scores after the contraction.
__local uint sh_q [KQ_TN*KQ_DKC/4];
__local half sh_qd[KQ_TN][KQ_DKC/32];
__local float * p8_lds = (__local float *)sh_q;
const size_t qbase = (size_t)head*n_q;
float slope = 1.0f;
if (max_bias > 0.0f) {
const float base = head < n_head_log2 ? m0 : m1;
const int ex = head < n_head_log2 ? head + 1 : 2*(head - n_head_log2) + 1;
slope = pow(base, ex);
}
global const half * mrow = (global const half *)(mask + offset_mask);
const uint mstride = (uint)(mask_nb1 / 2);
const int pblk = p_pitch / 32;
for (int m = 0; m < KQ_MB; ++m) {
const int kv = kv0 + m*KQ_TM;
if (kv - lid >= n_kv) {
break; // uniform across the workgroup
}
float acc[KQ_TN];
#pragma unroll
for (int t = 0; t < KQ_TN; ++t) {
acc[t] = 0.0f;
}
const int kl = min(kv, n_kv - 1);
const size_t kbase = ((size_t)head_kv*n_kv + kl)*nu;
const size_t kdbas = ((size_t)head_kv*n_kv + kl)*nblk;
for (int c0 = 0; c0 < nblk; c0 += KQ_DKC/32) {
const int cb = min(KQ_DKC/32, nblk - c0);
barrier(CLK_LOCAL_MEM_FENCE);
for (int i = lid; i < KQ_TN*(KQ_DKC/16); i += KQ_WG) {
const int t = i / (KQ_DKC/16);
const int u = (i % (KQ_DKC/16)) * 4;
const int qi = qn0 + t;
uint4 v = (uint4)(0u);
if (qi < n_q && u < cb*8) {
v = vload4(0, &qq[(qbase + qi)*nu + c0*8 + u]);
}
vstore4(v, 0, &sh_q[t*(KQ_DKC/4) + u]);
}
for (int i = lid; i < KQ_TN*(KQ_DKC/32); i += KQ_WG) {
const int t = i / (KQ_DKC/32);
const int b = i % (KQ_DKC/32);
const int qi = qn0 + t;
sh_qd[t][b] = (qi < n_q && b < cb) ? qd[(qbase + qi)*nblk + c0 + b] : (half)0.0f;
}
barrier(CLK_LOCAL_MEM_FENCE);
for (int b = 0; b < cb; ++b) {
const uint4 w0 = vload4(0, &kq[kbase + (c0 + b)*8]);
const uint4 w1 = vload4(0, &kq[kbase + (c0 + b)*8 + 4]);
const float dks = (float)kd[kdbas + c0 + b];
#pragma unroll
for (int t = 0; t < KQ_TN; ++t) {
const uint4 a0 = vload4(0, &sh_q[t*(KQ_DKC/4) + b*8]);
const uint4 a1 = vload4(0, &sh_q[t*(KQ_DKC/4) + b*8 + 4]);
int raw = 0;
raw = dot_acc_sat_4x8packed_ss_int(w0.s0, a0.s0, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s1, a0.s1, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s2, a0.s2, raw);
raw = dot_acc_sat_4x8packed_ss_int(w0.s3, a0.s3, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s0, a1.s0, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s1, a1.s1, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s2, a1.s2, raw);
raw = dot_acc_sat_4x8packed_ss_int(w1.s3, a1.s3, raw);
acc[t] += dks * (float)sh_qd[t][b] * (float)raw;
}
}
}
// Finish reading the Q slice before writing scores to the same local memory.
const int kvblk0 = kv - lid;
barrier(CLK_LOCAL_MEM_FENCE);
#pragma unroll
for (int t = 0; t < KQ_TN; ++t) {
p8_lds[t*64 + lid] = acc[t];
}
barrier(CLK_LOCAL_MEM_FENCE);
{
const int j = lid >> 1;
const int b = lid & 1;
const int n = qn0 + j;
if (n < n_q) {
const int kvb = kvblk0 + b*32;
__local float * src = p8_lds + j*64 + b*32;
const global half * mp = mrow + (size_t)n*mstride + kvb;
float4 s = vload4(0, src)*scale + slope*convert_float4(vload4(0, mp));
vstore4(s, 0, src);
float amax = fmax(fmax(s.x, s.y), fmax(s.z, s.w));
for (int c = 1; c < 8; ++c) {
s = vload4(c, src)*scale + slope*convert_float4(vload4(c, mp));
vstore4(s, c, src);
amax = fmax(amax, fmax(fmax(s.x, s.y), fmax(s.z, s.w)));
}
const size_t prow = qbase + n;
const int blk = kvb >> 5;
global uchar * qdst = qp + prow*(size_t)p_pitch + (size_t)kvb;
float sum = 0.0f;
// finite sentinel: the program builds with -cl-finite-math-only
if (amax > -1.0e30f) {
for (int c = 0; c < 8; ++c) {
const float4 e = exp(vload4(c, src) - amax);
sum += (e.x + e.y) + (e.z + e.w);
vstore4(convert_uchar4_sat_rte(e*255.0f), c, qdst);
}
} else {
for (int c = 0; c < 8; ++c) {
vstore4((uchar4)(0), c, qdst);
}
}
bmax[prow*pblk + blk] = amax;
bsum[prow*pblk + blk] = sum;
}
}
}
}
+71 -8
View File
@@ -107,7 +107,8 @@ GgmlOvDecoder::GgmlOvDecoder(ggml_cgraph * cgraph, std::map<std::string, std::sh
m_naive = true;
for (int i = 0; i < cgraph->n_nodes; ++i) {
if ((cgraph->nodes[i]->op == GGML_OP_GATED_DELTA_NET && cgraph->nodes[i]->op_params[0] > 1) ||
is_sampling_view(cgraph->nodes[i]) || is_conv_state_snapshot_writeback(cgraph->nodes[i])) {
is_sampling_view(cgraph->nodes[i]) || is_conv_state_snapshot_writeback(cgraph->nodes[i]) ||
is_ssm_state_writeback(cgraph->nodes[i])) {
std::tie(m_model_params, m_compute_params) = compute_llm_params(cgraph, false);
add_extra_inputs();
break;
@@ -494,6 +495,10 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
}
} else if (node->ne[0] == 1 && src->ne[0] * src->ne[1] * src->ne[2] == node->ne[1]) {
op_case = 3;
} else if (node->ne[0] * node->ne[1] == src->ne[0] * src->ne[1] && node->ne[2] == src->ne[2] &&
node->ne[3] == src->ne[3]) {
// head regroup, e.g. mamba2's [head_dim, n_head, n_tokens, n_seqs] -> [d_inner/n_group, n_group, n_tokens, n_seqs]
op_case = 1;
} else if (name.find("linear_attn_qkv_mixed") == 0 || name.find("alpha") == 0) {
op_case = 6;
} else if (name.find("linear_attn_out") == 0) {
@@ -597,6 +602,10 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
op_case = 3;
break;
}
if (is_column_set_rows(node)) {
op_case = 4;
break;
}
// `-fa off` writes the V cache transposed, through a [1, n] reshape of the cache: one element per row
if (node->src[2] != nullptr && node->src[2]->op == GGML_OP_RESHAPE && node->src[2]->ne[0] == 1 &&
is_kvcache(node->src[2]->src[0], nullptr)) {
@@ -797,6 +806,9 @@ int GgmlOvDecoder::compute_op_case(const ggml_tensor * node) const {
} else {
op_case = 1;
}
} else if (is_ssm_state_writeback(node)) {
op_case = 13;
break;
} else if (GgmlOvDecoder::is_conv_state_writeback(node)) {
op_case = 2;
break;
@@ -1296,6 +1308,11 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
writeback.snapshot_stride = destination->nb[2] / destination->nb[1];
compute_params.rs_writebacks[get_tensor_ov_name(cgraph, node)] = writeback;
}
if (is_ssm_state_writeback(node)) {
ComputeParams::RsWriteback writeback;
writeback.slot_begin = (int64_t) (node->src[1]->view_offs / node->src[1]->nb[1]);
compute_params.rs_writebacks[get_tensor_ov_name(cgraph, node)] = writeback;
}
if (is_conv_state_snapshot_writeback(node)) {
const auto source = node->src[0];
const auto dest = node->src[2];
@@ -1406,6 +1423,14 @@ std::pair<ModelParams, ComputeParams> GgmlOvDecoder::compute_llm_params(ggml_cgr
compute_params.attention_size =
ggml_openvino_static_kv_window(compute_params.attention_size, model_params.ctx_per_seq);
}
// The GPU plugin keeps device memory for each KV length it sees, and llama.cpp grows the KV length in steps of
// 256. Round it up to a power of two the same way, so that a long context uses a few lengths and not one per
// step. The KQ mask input is padded to this length with -inf (convert_ggml_input_to_ov).
if (!is_static && ggml_openvino_is_gpu() && !ggml_openvino_getenv_int("GGML_OPENVINO_STATEFUL_EXECUTION") &&
!compute_params.mask_name.empty() && compute_params.attention_size > 0 && model_params.ctx_per_seq > 0) {
compute_params.attention_size =
ggml_openvino_static_kv_window(compute_params.attention_size, model_params.ctx_per_seq);
}
model_params.ctx = model_params.ctx_per_seq * model_params.n_seq;
return {model_params, compute_params};
}
@@ -1416,6 +1441,15 @@ void GgmlOvDecoder::validate_cgraph() const {
}
}
bool GgmlOvDecoder::has_inp_embd_leaf_use(const ggml_tensor * tensor) const {
for (int i = 0; i < m_cgraph->n_nodes; i++) {
if (is_inp_embd_leaf(tensor, m_cgraph->nodes[i])) {
return true;
}
}
return false;
}
ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
const ggml_tensor * input,
int dynamic_dim_index) const {
@@ -1530,7 +1564,7 @@ ov::PartialShape GgmlOvDecoder::get_graph_input_shape(const ggml_tensor * op,
// index list has a static length; on CPU/GPU it may change across compiles (defrag).
input_shape = m_is_static ? ov::PartialShape{get_shape(input)} : ov::PartialShape{1, 1, 1, -1};
} else if (input != nullptr && op != nullptr && is_inp_emb(input, op)) {
} else if (input != nullptr && op != nullptr && (is_inp_emb(input, op) || has_inp_embd_leaf_use(input))) {
input_shape = ov::PartialShape{get_shape(input)};
if (!m_is_static) {
input_shape[2] = -1;
@@ -1914,7 +1948,11 @@ void GgmlOvDecoder::compute_model_outputs() {
continue;
}
const bool inactive_top_k = is_sampling_top_k(cur_node) && !(cur_node->flags & GGML_TENSOR_FLAG_COMPUTE);
if ((cur_node->flags & GGML_TENSOR_FLAG_OUTPUT) || inactive_top_k) {
auto cur_node_use_count = m_cgraph->use_counts[ggml_hash_find(&m_cgraph->visited_hash_set, cur_node)];
// llama flags result_norm in every graph but reads it only with embeddings on, when it has no consumer
// and is kept by the rule below.
const bool unread_embd = strcmp(cur_node->name, "result_norm") == 0 && cur_node_use_count > 0;
if (((cur_node->flags & GGML_TENSOR_FLAG_OUTPUT) && !unread_embd) || inactive_top_k) {
const auto name = get_tensor_ov_name(m_cgraph, cur_node);
m_model_outputs[name] = cur_node;
m_model_output_names.insert(name);
@@ -1925,7 +1963,12 @@ void GgmlOvDecoder::compute_model_outputs() {
m_model_outputs[name] = cache;
m_model_output_names.insert(name);
}
auto cur_node_use_count = m_cgraph->use_counts[ggml_hash_find(&m_cgraph->visited_hash_set, cur_node)];
if (is_ssm_state_writeback(cur_node)) {
auto * cache = cur_node->view_src;
const auto name = get_tensor_ov_name(m_cgraph, cache);
m_model_outputs[name] = cache;
m_model_output_names.insert(name);
}
if (cur_node_use_count == 0) {
// The output of in-place ops is the view_src tensor, which is updated in place. We should use the view_src name as the output name to make sure it can be correctly matched with the later ops that use the view_src.
if (cur_node != nullptr && ::is_inplace_op(cur_node) &&
@@ -2641,6 +2684,10 @@ bool GgmlOvDecoder::is_direct_gdn_view(const ggml_tensor * node) const {
std::vector<std::string> GgmlOvDecoder::get_output_names(int node_idx) const {
auto * node = m_node_info_list[node_idx].node;
if (is_ssm_state_writeback(node)) {
// the written state block plus the full updated cache (see translate_cpy op_case 13)
return {m_node_info_list[node_idx].node_name, get_tensor_ov_name(m_cgraph, node->view_src)};
}
if (node->op == GGML_OP_GATED_DELTA_NET && (!m_model_params.has_rs_rollback || node->op_params[0] > 1)) {
std::string attn_name;
std::string state_name;
@@ -2669,6 +2716,10 @@ std::vector<std::string> GgmlOvDecoder::get_output_names(int node_idx) const {
std::string GgmlOvDecoder::get_inplace_op_src(int node_idx) const {
auto * node = m_node_info_list[node_idx].node;
if (is_ssm_state_writeback(node)) {
// translate_cpy publishes the updated cache as a second named output
return "";
}
// Reset lengths (also in static mode) and dynamic remainder views can become nonempty
// after compilation. Keep their alias updates in the translated cache chain.
const bool runtime_recurrent_write = !m_naive && is_recurrent_cache(node->view_src) &&
@@ -2786,7 +2837,7 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
m_node_dynamic_dims[src] = 0;
continue;
}
if (is_inp_scale_rows(src, node)) {
if (is_inp_scale_rows(src, node) || is_inp_embd_leaf(src, node)) {
m_node_dynamic_dims[src] = 1;
continue;
}
@@ -2865,6 +2916,11 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
m_node_dynamic_dims[node] = 2;
break;
}
if (node->src[0]->op == GGML_OP_SSM_SCAN) {
// y part [head_dim, n_head, n_tokens, n_seqs] at offset 0; the state part is token-independent
m_node_dynamic_dims[node] = node->view_offs == 0 ? 2 : -1;
break;
}
// Use stride-based matching: the stride of a VIEW dimension directly
// encodes which source dimension it indexes into, so it uniquely
// identifies the dynamic dim even when two dims share the same size.
@@ -3030,6 +3086,10 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
case GGML_OP_GATED_DELTA_NET:
m_node_dynamic_dims[node] = 1;
break;
case GGML_OP_SSM_SCAN:
// packed 1-D [y | state]; the y part grows with tokens
m_node_dynamic_dims[node] = 0;
break;
case GGML_OP_RMS_NORM:
case GGML_OP_L2_NORM:
case GGML_OP_NORM:
@@ -3080,12 +3140,15 @@ void GgmlOvDecoder::compute_node_dynamic_dims() {
case GGML_OP_CPY:
case GGML_OP_SUM:
case GGML_OP_MEAN:
case GGML_OP_ARANGE:
m_node_dynamic_dims[node] = -1;
break;
case GGML_OP_SET_ROWS:
// a cache write returns the cache; other rows writes keep the shape of their destination
m_node_dynamic_dims[node] = (is_kvcache_write(node) || is_conv_state_snapshot_writeback(node)) ?
-1 : m_node_dynamic_dims[node->src[2]];
// a cache or column write returns its whole buffer; other rows writes keep the shape of their destination
m_node_dynamic_dims[node] =
(is_kvcache_write(node) || is_conv_state_snapshot_writeback(node) || is_column_set_rows(node)) ?
-1 :
m_node_dynamic_dims[node->src[2]];
break;
case GGML_OP_IM2COL: {
m_node_dynamic_dims[node] = -1;
+61 -5
View File
@@ -477,6 +477,21 @@ public:
return tensor->op == GGML_OP_GET_ROWS && op->op == GGML_OP_RMS_NORM;
}
// Raw float input [n_embd, n_tokens]: DFlash/DSpark/EAGLE3 features into fc, or raw embd batch / EAGLE3 g-embd into RMS_NORM
// MUL_MAT needs a weight leaf as src[0], so inp_mean (src[0] is computed) does not match
static bool is_inp_embd_leaf(const ggml_tensor * tensor, const ggml_tensor * op) {
if (tensor->op != GGML_OP_NONE || !(tensor->flags & GGML_TENSOR_FLAG_INPUT) || tensor->type != GGML_TYPE_F32) {
return false;
}
if (op->op == GGML_OP_MUL_MAT) {
return op->src[1] == tensor && op->src[0] != nullptr && op->src[0]->op == GGML_OP_NONE;
}
return op->op == GGML_OP_RMS_NORM && op->src[0] == tensor;
}
// true if any node of the current graph uses tensor as a raw embd input (see is_inp_embd_leaf)
bool has_inp_embd_leaf_use(const ggml_tensor * tensor) const;
static bool is_activation_input(const ggml_tensor * tensor, const ggml_tensor * op) {
if (tensor == nullptr || op == nullptr || tensor->type != GGML_TYPE_F32 || tensor->ne[2] != 1 ||
tensor->ne[3] != 1 || (tensor->buffer != nullptr && tensor->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS)) {
@@ -578,11 +593,11 @@ public:
bool is_direct_gdn_view(const ggml_tensor * node) const;
inline static bool is_cache(const ggml_tensor * tensor, const ggml_tensor * op) {
static bool is_cache(const ggml_tensor * tensor, const ggml_tensor * op) {
return is_recurrent_cache(tensor) || is_kvcache(tensor, op);
}
inline static bool is_kvcache(const ggml_tensor * tensor, const ggml_tensor * op) {
static bool is_kvcache(const ggml_tensor * tensor, const ggml_tensor * op) {
if (tensor == nullptr || is_recurrent_cache(tensor)) {
return false;
}
@@ -590,27 +605,68 @@ public:
(op != nullptr && op->op == GGML_OP_SET_ROWS && op->src[2] == tensor);
}
inline static bool is_conv_state_writeback(const ggml_tensor * node) {
static bool is_conv_state_writeback(const ggml_tensor * node) {
return node->op == GGML_OP_CPY && node->view_src != nullptr && is_recurrent_cache(node->view_src) &&
node->src[0] != nullptr && node->src[0]->op == GGML_OP_VIEW && node->src[0]->src[0] != nullptr &&
node->src[0]->src[0]->op == GGML_OP_CONCAT && node->src[1] != nullptr &&
node->src[1]->op == GGML_OP_VIEW && node->src[1]->view_src == node->view_src;
}
// The final-state writeback of SSM_SCAN (K == 1): the state tail of the packed [y | state]
// output copied into a slot block of the recurrent cache.
static bool is_ssm_state_writeback(const ggml_tensor * node) {
if (!(node != nullptr && node->op == GGML_OP_CPY && node->view_src != nullptr && node->src[0] != nullptr &&
node->src[0]->op == GGML_OP_VIEW && node->src[0]->src[0] != nullptr &&
node->src[0]->src[0]->op == GGML_OP_SSM_SCAN && node->src[0]->src[0]->op_params[0] == 1 &&
node->src[1] != nullptr && node->src[1]->op == GGML_OP_VIEW &&
node->src[1]->view_src == node->view_src)) {
return false;
}
const auto * scan = node->src[0]->src[0];
const auto * src = node->src[0];
const auto * dst = node->src[1];
if (node->type != GGML_TYPE_F32 || src->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32 ||
src->ne[2] != 1 || src->ne[3] != 1) {
return false;
}
for (int i = 0; i < GGML_MAX_DIMS; ++i) {
if (src->ne[i] != dst->ne[i]) {
return false;
}
}
return dst->nb[1] > 0 && dst->view_offs % dst->nb[1] == 0 &&
src->view_offs == (size_t) ggml_nelements(scan->src[1]) * ggml_type_size(scan->type);
}
// build_conv_state_store writes overlapping [history, window, channel, seq] windows into newest-first rollback planes.
// Match this layout separately from general SET_ROWS.
static bool is_conv_state_snapshot_writeback(const ggml_tensor * node);
// SET_ROWS into a column range of each row of a plain 2-D buffer, e.g. build_layer_inp_dev(): every enabled layer
// input goes to view_2d(layer_inp_dev, n_embd, n_batch, nb1, col * n_embd * elem) at the batch rows of the ubatch.
// The buffer sits in a regular device buffer, so is_kvcache() matches it: check this first
static bool is_column_set_rows(const ggml_tensor * node) {
if (node == nullptr || node->op != GGML_OP_SET_ROWS || node->src[2] == nullptr ||
node->src[2]->op != GGML_OP_VIEW) {
return false;
}
const ggml_tensor * dst = node->src[2];
const ggml_tensor * buf = dst->view_src;
return buf != nullptr && buf->op == GGML_OP_NONE && !is_recurrent_cache(buf) && buf->ne[2] == 1 &&
buf->ne[3] == 1 && dst->ne[2] == 1 && dst->ne[3] == 1 && dst->ne[0] < buf->ne[0] &&
dst->nb[0] == buf->nb[0] && dst->nb[1] == buf->nb[1] && dst->ne[1] == buf->ne[1];
}
// SET_ROWS into a KV cache, directly or through the [1, n] reshape of the transposed V cache with `-fa off`.
// Other SET_ROWS write rows of a compute tensor, e.g. the MoE expert-group mask
inline static bool is_kvcache_write(const ggml_tensor * node) {
static bool is_kvcache_write(const ggml_tensor * node) {
const ggml_tensor * dst = node->src[2];
return dst != nullptr && is_kvcache(dst->view_src != nullptr ? dst->view_src : dst, nullptr);
}
// GET_ROWS batched over ggml dims 2 and 3, e.g. the MoE expert-group scores
// ggml_get_rows(ggml_reshape_4d(groups, 1, n_exp_per_group, n_groups, n_tokens), top_groups)
inline static bool is_get_rows_batched_4d(const ggml_tensor * node) {
static bool is_get_rows_batched_4d(const ggml_tensor * node) {
return node->op == GGML_OP_GET_ROWS && !ggml_is_quantized(node->src[0]->type) &&
node->src[1]->ne[1] == node->src[0]->ne[2] && node->src[1]->ne[2] == node->src[0]->ne[3] &&
node->src[1]->ne[3] == 1;
@@ -616,6 +616,10 @@ bool ggml_openvino_is_npu() {
return ggml_openvino_get_device_config().is_npu;
}
bool ggml_openvino_is_static() {
return ggml_openvino_is_npu() || ggml_openvino_getenv_int("GGML_OPENVINO_FORCE_STATIC");
}
bool ggml_openvino_is_gpu() {
return has_prefix(ggml_openvino_get_device_name(), "GPU");
}
@@ -134,6 +134,9 @@ bool ggml_openvino_release_weights_enabled();
// Check if running on NPU
bool ggml_openvino_is_npu();
// Static graphs: NPU, or any device with GGML_OPENVINO_FORCE_STATIC
bool ggml_openvino_is_static();
// KV-cache slicing: bind the KV cache, and size the KQ masks, at the attended length instead of the full context.
// One mechanism, two modes:
// dynamic graphs (CPU/GPU): the KV input is a view over the cache; the graph is not recompiled.
+56 -5
View File
@@ -1504,6 +1504,38 @@ static bool has_strides_on_element_grid(const ggml_tensor * t) {
return true;
}
// A VIEW that regroups one contiguous span of every row of its contiguous source, e.g. the z/x/B/C
// splits of the fused mamba projections. The strides do not nest on an element grid, but
// process_view_input_new resolves it as a Slice of the source's innermost axis plus a Reshape.
// Keep in sync with the matching branch there; dynamic execution only.
static bool is_row_span_view(const ggml_tensor * t) {
if (ggml_openvino_is_static() || is_stateful_enabled()) {
return false;
}
const ggml_tensor * src = t->op == GGML_OP_VIEW ? t->src[0] : nullptr;
// a RESHAPE source is fine (the translation reads its value); a chained VIEW is not, because
// the view metadata then describes the intermediate view instead of the resolved input
if (src == nullptr || src->op == GGML_OP_VIEW || !ggml_is_contiguous(src) || t->type != src->type) {
return false;
}
const size_t es = ggml_type_size(t->type);
int d = 0;
size_t span = 1;
while (d < GGML_MAX_DIMS && t->nb[d] == span * es) {
span *= t->ne[d];
d++;
}
if (d < 2) {
return false;
}
for (int s = 1; d < GGML_MAX_DIMS; ++d, ++s) {
if (t->ne[d] != src->ne[s] || t->nb[d] != src->nb[s]) {
return false;
}
}
return t->view_offs % es == 0 && t->view_offs / es + span <= (size_t) src->ne[0];
}
static bool has_non_contiguous_view_input(const ggml_tensor * op) {
for (int i = 0; i < GGML_MAX_SRC; i++) {
if (op->src[i] == nullptr) {
@@ -1624,7 +1656,7 @@ static bool cpy_output_view_is_supported(const ggml_tensor * op) {
}
return ggml_nbytes(op) == 0 || ggml_is_contiguous(op) || GgmlOvDecoder::is_conv_state_writeback(op) ||
GgmlOvDecoder::is_gdn_snapshot_writeback(op);
GgmlOvDecoder::is_gdn_snapshot_writeback(op) || GgmlOvDecoder::is_ssm_state_writeback(op);
}
static bool mul_mat_id_requires_large_tmp(const ggml_tensor * op) {
@@ -1727,14 +1759,16 @@ static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
}
case GGML_OP_GET_ROWS:
case GGML_OP_SET_ROWS: {
if (GgmlOvDecoder::is_conv_state_snapshot_writeback(op) &&
(ggml_openvino_is_npu() || ggml_openvino_getenv_int("GGML_OPENVINO_FORCE_STATIC"))) {
if (GgmlOvDecoder::is_conv_state_snapshot_writeback(op) && ggml_openvino_is_static()) {
return {false, "convolution snapshots require dynamic execution"};
}
if (op->ne[3] != 1 && !GgmlOvDecoder::is_get_rows_batched_4d(op) &&
!GgmlOvDecoder::is_conv_state_snapshot_writeback(op)) {
return {false, "GET_ROWS/SET_ROWS with ne[3] != 1 (ne[3]=" + std::to_string(op->ne[3]) + ") is not supported"};
}
if (op->op == GGML_OP_SET_ROWS && op->src[0]->type == GGML_TYPE_BF16) {
return {false, "SET_ROWS with BF16 src0 is not supported"};
}
if (op->ne[0] == 256 && (op->src[0]->type == GGML_TYPE_Q4_K || op->src[0]->type == GGML_TYPE_Q5_K ||
op->src[0]->type == GGML_TYPE_Q4_1 || op->src[0]->type == GGML_TYPE_Q5_1)) {
// These are all f16-arithmetic dequant rounding errors that intermittently exceed the
@@ -2004,6 +2038,9 @@ static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
if (op->view_src != nullptr && !ggml_is_contiguous(op->src[0])) {
return {false, "ROPE on VIEW / non-contiguous input is not supported"};
}
if (op->type == GGML_TYPE_BF16) {
return {false, "ROPE with BF16 type is not supported"};
}
break;
}
case GGML_OP_TRANSPOSE: {
@@ -2034,7 +2071,7 @@ static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
reason = "GATED_DELTA_NET with kda (per-key-dimension gating) is not supported";
}
else if (op->op_params[0] > 1) {
if (ggml_openvino_is_npu() || ggml_openvino_getenv_int("GGML_OPENVINO_FORCE_STATIC")) {
if (ggml_openvino_is_static()) {
reason = "GATED_DELTA_NET snapshots require dynamic execution";
}
}
@@ -2053,6 +2090,19 @@ static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
// return true;
break;
}
case GGML_OP_SSM_SCAN: {
// SelectiveSSM implements the Mamba-2 recurrence: scalar decay per head, one final state
if (op->src[3]->ne[0] != 1) {
return {false, "SSM_SCAN with Mamba-1 state-wise decay is not supported"};
}
if (op->op_params[0] > 1) {
return {false, "SSM_SCAN with rollback snapshots (K > 1) is not supported"};
}
if (ggml_openvino_is_static()) {
return {false, "SSM_SCAN requires dynamic execution"};
}
break;
}
case GGML_OP_VIEW: {
// Skip TOPK_MOE fused tests until it is fully supported.
// The argsort_top_k VIEW wrapping ARGSORT is named "selected_experts" in test_topk_moe.
@@ -2274,7 +2324,8 @@ static ggml_openvino_op_support ggml_backend_openvino_device_supports_op_impl(gg
if (supported_types.find(src->type) == supported_types.end()) {
return {false, "src[" + std::to_string(i) + "] type " + std::string(ggml_type_name(src->type)) + " is not supported"};
}
if (!has_strides_on_element_grid(src) && !(i == 1 && GgmlOvDecoder::is_gdn_snapshot_writeback(op)) &&
if (!has_strides_on_element_grid(src) && !is_row_span_view(src) &&
!(i == 1 && GgmlOvDecoder::is_gdn_snapshot_writeback(op)) &&
!(i == 2 && GgmlOvDecoder::is_conv_state_snapshot_writeback(op))) {
return {false, "src[" + std::to_string(i) + "] strides are not multiples of each other"};
}
+5 -3
View File
@@ -197,8 +197,7 @@ void ggml_openvino_model_cache_init() {
}
return;
}
if (cache_only && (ggml_openvino_is_npu() || ggml_openvino_getenv_int("GGML_OPENVINO_FORCE_STATIC") ||
ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_CACHE") ||
if (cache_only && (ggml_openvino_is_static() || ggml_openvino_getenv_int("GGML_OPENVINO_DISABLE_CACHE") ||
ggml_openvino_getenv_int("GGML_OPENVINO_ENABLE_FALLBACK"))) {
GGML_ABORT("ggml-openvino: cache-only mode requires dynamic CPU/GPU execution with caching and without fallback");
}
@@ -368,7 +367,8 @@ uint64_t ggml_openvino_model_fingerprint(const ggml_cgraph * cgraph,
// Version 5 requantizes K-quant MoE experts to Q4_1_64 on GPU by default.
// Version 6 clears recurrent-cache slots with Select, so a NaN in a cleared slot does not survive.
// Version 7 reads ROPE frequency factors named other than rope_freqs.weight as a Constant, not a Parameter.
h = fnv1a_u64(h, 7);
// Version 8 drops result_norm from the outputs when it has a consumer, and hashes the output flags.
h = fnv1a_u64(h, 8);
h = fnv1a(h, graph_signature.data(), graph_signature.size());
for (const char * name : cache_settings) {
const char * value = ggml_openvino_getenv_str(name, "");
@@ -391,6 +391,8 @@ uint64_t ggml_openvino_model_fingerprint(const ggml_cgraph * cgraph,
const ggml_tensor * node = cgraph->nodes[i];
h = fnv1a_u64(h, static_cast<uint64_t>(node->op));
h = fnv1a(h, node->name, strlen(node->name));
// Each output is a Result of the model; llama_set_embeddings_layer_inp() changes only the flags.
h = fnv1a_u64(h, (node->flags & GGML_TENSOR_FLAG_OUTPUT) ? 1u : 0u);
has_gdn = has_gdn || node->op == GGML_OP_GATED_DELTA_NET;
}
if (has_gdn && ggml_openvino_is_npu()) {
@@ -75,6 +75,28 @@ OutputVector translate_cpy(const NodeContext & context) {
return rename_outputs_with_suffix({result}, context.get_name());
}
if (op_case == 13) {
// SSM_SCAN final-state writeback: output the written state block and the full updated cache
auto src = process_view_input_new(context, 0);
auto base = context.get_input(1);
if (src.get_element_type() != context.get_output_type()) {
src = std::make_shared<ov::op::v0::Convert>(src, context.get_output_type());
}
auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0});
auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1});
auto int_max = ov::op::v0::Constant::create(ov::element::i64, {1}, {INT_MAX});
auto axis = ov::op::v0::Constant::create(ov::element::i64, {1}, {2});
auto begin = context.get_input("rs_slot_begin_" + context.get_name());
auto src_len = std::make_shared<ov::op::v8::Gather>(
std::make_shared<ov::op::v3::ShapeOf>(src, ov::element::i64), axis,
ov::op::v0::Constant::create(ov::element::i64, {}, {0}));
auto end = std::make_shared<ov::op::v1::Add>(begin, src_len);
auto head_part = std::make_shared<ov::op::v8::Slice>(base, zero, begin, one, axis);
auto tail_part = std::make_shared<ov::op::v8::Slice>(base, end, int_max, one, axis);
auto updated = std::make_shared<ov::op::v0::Concat>(ov::OutputVector{head_part, src, tail_part}, 2);
return rename_outputs_with_suffix({src, updated}, context.get_name());
}
if (op_case == 4) {
auto src = process_view_input_new(context, 0);
auto base = context.get_input(1);
+12 -6
View File
@@ -4,6 +4,7 @@
#include <openvino/op/add.hpp>
#include <openvino/op/constant.hpp>
#include <openvino/op/convert.hpp>
#include <openvino/op/multiply.hpp>
#include <vector>
@@ -26,15 +27,20 @@ OutputVector translate_scale(const NodeContext & context) {
return translate_recurrent_cache_reset(context);
}
auto scaled = std::make_shared<ov::op::v1::Multiply>(process_view_input_new(context, 0), scale_node);
std::shared_ptr<ov::Node> res;
// Multiply and Add need matching types; compute f16 input in f32 and round once, as the CPU backend does.
auto input = process_view_input_new(context, 0);
const auto input_type = input.get_element_type();
if (input_type != ov::element::f32) {
input = std::make_shared<ov::op::v0::Convert>(input, ov::element::f32);
}
ov::Output<ov::Node> res = std::make_shared<ov::op::v1::Multiply>(input, scale_node);
if (bias != 0.0f) {
auto bias_node =
std::make_shared<ov::op::v0::Constant>(ov::element::f32, ov::Shape{}, std::vector<float>{bias});
res = std::make_shared<ov::op::v1::Add>(scaled, bias_node);
} else {
res = scaled;
res = std::make_shared<ov::op::v1::Add>(res, bias_node);
}
if (input_type != ov::element::f32) {
res = std::make_shared<ov::op::v0::Convert>(res, input_type);
}
return rename_outputs_with_suffix({res}, context.get_name());
@@ -0,0 +1,42 @@
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
// Local mirror of OpenVINO's ov::op::internal::SelectiveSSM declaration
// (src/core/dev_api/openvino/op/selective_ssm.hpp). The implementation comes from the linked
// OpenVINO runtime; the class layout must stay in sync with the runtime's declaration.
#pragma once
#include "openvino/op/op.hpp"
namespace ov::op::internal {
/// Mamba2 selective state-space recurrence (arXiv:2405.21060).
/// dA = exp(A * dt), dtB = dt * B; per token t:
/// dBx_t = x_t (x) dtB_t, state_t = state_{t-1} * dA_t + dBx_t, y_t = sum(state_t * C_t).
/// B and C are grouped and shared across heads: head h reads group h / (num_heads / num_groups).
class OPENVINO_API SelectiveSSM : public ov::op::Op {
public:
OPENVINO_OP("SelectiveSSM")
SelectiveSSM() = default;
/// \param A (Negative) log-decay rates per head [num_heads].
/// \param dt Per-token, per-head time steps [batch_size, seq_len, num_heads].
/// \param B Grouped input projection [batch_size, seq_len, num_groups, state_size].
/// \param x Input hidden states [batch_size, seq_len, num_heads, head_dim].
/// \param C Grouped output projection [batch_size, seq_len, num_groups, state_size].
/// \param recurrent_state Initial state [batch_size, num_heads, head_dim, state_size].
SelectiveSSM(const Output<Node> & A,
const Output<Node> & dt,
const Output<Node> & B,
const Output<Node> & x,
const Output<Node> & C,
const Output<Node> & recurrent_state);
/// \param args Input tensor vector in order: A, dt, B, x, C, recurrent_state.
explicit SelectiveSSM(const ov::OutputVector & args);
void validate_and_infer_types() override;
bool visit_attributes(AttributeVisitor & visitor) override;
std::shared_ptr<ov::Node> clone_with_new_inputs(const ov::OutputVector & new_args) const override;
};
} // namespace ov::op::internal
@@ -90,12 +90,71 @@ static OutputVector translate_conv_snapshots(const NodeContext & context) {
return rename_outputs_with_suffix({result}, name);
}
// SET_ROWS into a column range of a plain 2-D buffer (see GgmlOvDecoder::is_column_set_rows). The result is the
// whole buffer: the indexed rows get the new columns, the other columns and rows keep their values, so the next
// write into another column range of the same buffer reads this one.
static OutputVector translate_column_set_rows(const NodeContext & context) {
using namespace ov::op;
auto i64_const = [](const std::vector<int64_t> & values) {
return v0::Constant::create(ov::element::i64, {values.size()}, values);
};
const auto view_shape = context.get_view_input_ggml_shape(2, 0); // [1, 1, rows, width]
FRONT_END_OP_CONVERSION_CHECK(view_shape.size() == 4, "SET_ROWS column write needs a 4-D destination view");
const int64_t width = (int64_t) view_shape[3];
auto base = context.get_input(2);
const auto base_shape = base.get_partial_shape();
FRONT_END_OP_CONVERSION_CHECK(base_shape.rank().is_static() && base_shape[base_shape.size() - 1].is_static(),
"SET_ROWS column write needs a static buffer row size");
const int64_t row_size = base_shape[base_shape.size() - 1].get_length();
const size_t elem_size = context.get_output_type().size();
const int64_t col_begin = (int64_t) (context.get_view_input_offset(2, 0) / elem_size) % row_size;
const int64_t col_end = col_begin + width;
FRONT_END_OP_CONVERSION_CHECK(col_end <= row_size, "SET_ROWS column range is out of bounds");
Output<Node> data = std::make_shared<v1::Reshape>(process_view_input_new(context, 0), i64_const({1, 1, -1, width}),
false);
if (data.get_element_type() != context.get_output_type()) {
data = std::make_shared<v0::Convert>(data, context.get_output_type());
}
Output<Node> rows = std::make_shared<v1::Reshape>(process_view_input_new(context, 1), i64_const({-1}), false);
if (rows.get_element_type() != ov::element::i64) {
rows = std::make_shared<v0::Convert>(rows, ov::element::i64);
}
auto axis = v0::Constant::create(ov::element::i64, {}, {2});
auto buffer = std::make_shared<v1::Reshape>(base, i64_const({1, 1, -1, row_size}), false);
Output<Node> updates = data;
if (width < row_size) {
auto existing = std::make_shared<v8::Gather>(buffer, rows, axis);
auto col_axis = i64_const({3});
OutputVector parts;
if (col_begin > 0) {
parts.push_back(std::make_shared<v8::Slice>(existing, i64_const({0}), i64_const({col_begin}),
i64_const({1}), col_axis));
}
parts.push_back(data);
if (col_end < row_size) {
parts.push_back(std::make_shared<v8::Slice>(existing, i64_const({col_end}), i64_const({row_size}),
i64_const({1}), col_axis));
}
updates = std::make_shared<v0::Concat>(parts, 3);
}
auto updated = std::make_shared<v3::ScatterUpdate>(buffer, rows, updates, axis);
auto res = std::make_shared<v1::Reshape>(updated, std::make_shared<v3::ShapeOf>(base, ov::element::i64), false);
return rename_outputs_with_suffix({res}, context.get_name());
}
OutputVector translate_set_rows(const NodeContext & context) {
num_inputs_check(context, 3, 3);
if (context.get_op_case() == 3) {
return translate_conv_snapshots(context);
}
if (context.get_op_case() == 4) {
return translate_column_set_rows(context);
}
auto data = process_view_input_new(context, 0);
auto indices = process_view_input_new(context, 1);
@@ -0,0 +1,63 @@
// Copyright (C) 2018-2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0
//
// Adapted from OpenVINO's GGUF frontend (src/frontends/gguf/src/op/ssm_scan.cpp).
#include "../node_context.h"
#include "../op_table.h"
#include "../utils.h"
#include "selective_ssm.hpp"
#include <openvino/op/concat.hpp>
#include <openvino/op/constant.hpp>
#include <openvino/op/gather.hpp>
#include <openvino/op/reshape.hpp>
#include <openvino/op/shape_of.hpp>
#include <openvino/op/softplus.hpp>
namespace ov {
namespace frontend {
namespace ggml {
namespace op {
OutputVector translate_ssm_scan(const NodeContext & context) {
num_inputs_check(context, 7, 7);
using namespace ov::op;
// ggml: s [slots,H,D,S], x [B,T,H,D], dt [1,B,T,H], A [1,1,H,1],
// B/C [B,T,G,S], ids [1,1,1,B]. Only Mamba2 has scalar decay per head.
const auto a_shape = context.get_input_shape(3);
FRONT_END_OP_CONVERSION_CHECK(a_shape.rank() == 4 && a_shape[0] == 1 && a_shape[1] == 1 && a_shape[3] == 1,
"SSM_SCAN: SelectiveSSM requires Mamba2 scalar decay per head; "
"Mamba1 state-wise decay is not supported");
const auto x = process_view_input_new(context, 1);
const auto flat = v0::Constant::create(element::i64, {1}, {-1});
const auto axis0 = v0::Constant::create(element::i64, {}, {0});
const auto ids = std::make_shared<v1::Reshape>(process_view_input_new(context, 6), flat, false);
const auto state = std::make_shared<v8::Gather>(process_view_input_new(context, 0), ids, axis0);
const auto a = std::make_shared<v1::Reshape>(process_view_input_new(context, 3), flat, false);
const auto dt_shape = std::make_shared<v8::Gather>(std::make_shared<v3::ShapeOf>(x),
v0::Constant::create(element::i64, {3}, {0, 1, 2}),
axis0);
const auto dt =
std::make_shared<v4::SoftPlus>(std::make_shared<v1::Reshape>(process_view_input_new(context, 2), dt_shape, false));
const auto ssm = std::make_shared<internal::SelectiveSSM>(a,
dt,
process_view_input_new(context, 4),
x,
process_view_input_new(context, 5),
state);
// Pack directly in GGML layout so whole-part views can select either Concat input.
const auto packed_shape = v0::Constant::create(element::i64, {4}, {1, 1, 1, -1});
const auto y = std::make_shared<v1::Reshape>(ssm->output(0), packed_shape, false);
const auto s = std::make_shared<v1::Reshape>(ssm->output(1), packed_shape, false);
const auto result = std::make_shared<v0::Concat>(OutputVector{y, s}, 3);
result->get_rt_info()[GGML_OV_SSM_SCAN_PACKED] = true;
return rename_outputs_with_suffix({result}, context.get_name());
}
} // namespace op
} // namespace ggml
} // namespace frontend
} // namespace ov
+2 -1
View File
@@ -70,7 +70,8 @@ OutputVector translate_tri(const NodeContext & context) {
throw std::runtime_error("translate_tri: invalid tri_type " + std::to_string(tri_type));
}
auto zero = ov::op::v0::Constant::create(ov::element::f32, {}, {0.0f});
// Select needs the zero in the input's type (f32, f16 or bf16).
auto zero = ov::op::v0::Constant::create(x.get_element_type(), {}, {0.0f});
auto res = std::make_shared<ov::op::v1::Select>(mask, x, zero);
return rename_outputs_with_suffix({res}, context.get_name());
@@ -98,6 +98,7 @@ std::unordered_map<std::string, CreatorFunction> get_supported_ops() {
{"GGML_OP_PAD", op::translate_pad },
{"GGML_OP_TOP_K", op::translate_top_k },
{"GGML_OP_SSM_CONV", op::translate_ssm_conv },
{"GGML_OP_SSM_SCAN", op::translate_ssm_scan },
{"GGML_OP_GATED_DELTA_NET", op::translate_gated_delta_net },
{"GGML_OP_REPEAT", op::translate_repeat },
{"GGML_OP_CUMSUM", op::translate_cumsum },
@@ -49,6 +49,7 @@ GGML_OP_CONVERTER(translate_flash_attn_ext);
GGML_OP_CONVERTER(translate_clamp);
GGML_OP_CONVERTER(translate_pad);
GGML_OP_CONVERTER(translate_ssm_conv);
GGML_OP_CONVERTER(translate_ssm_scan);
GGML_OP_CONVERTER(translate_gated_delta_net);
#ifdef GGML_OPENVINO_HAS_CACHE_UPDATE
OutputVector translate_recurrent_gated_delta_net(const NodeContext & context, const OutputVector & cache_inputs);
+102 -11
View File
@@ -465,6 +465,25 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
return input;
}
// A VIEW over the packed SSM_SCAN output selects one whole Concat part: y at offset 0, the
// final state after it. The y length is token-dependent, so a static Slice of the flat
// tensor would bake in the captured token count; return the native part instead.
if (view_input_size == 1 && input.get_node_shared_ptr()->get_rt_info().count(GGML_OV_SSM_SCAN_PACKED)) {
const bool is_y = context.get_view_input_offset(input_index, 0) == 0;
auto part = input.get_node_shared_ptr()->input_value(is_y ? 0 : 1).get_node_shared_ptr()->input_value(0);
if (is_y) {
// y keeps its native [n_seqs, n_tokens, n_head, head_dim] layout
return part;
}
// state [n_seqs, n_head, head_dim, d_state] -> view layout [1, 1, n_seqs, d_state * d_inner]
auto n_seqs = get_dimensions(std::make_shared<ov::op::v3::ShapeOf>(part, ov::element::i64), {0});
auto target = std::make_shared<ov::op::v0::Concat>(
ov::OutputVector{ov::op::v0::Constant::create(ov::element::i64, {2}, {1, 1}), n_seqs,
ov::op::v0::Constant::create(ov::element::i64, {1}, {-1})},
0);
return std::make_shared<ov::op::v1::Reshape>(part, target, false);
}
// Integer router views can share these strides when their row count is zero.
// Packed projections can store query and gate in separate halves of each head.
if (!context.is_static() && view_input_size == 1 && input.get_element_type().is_real()) {
@@ -534,6 +553,70 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
}
}
// A VIEW that regroups one contiguous span of every source row, e.g. the z/x/B/C splits of the
// fused mamba projections: [head_dim, n_head, nt, ns] over [d_in_proj, nt, ns] rows. The span
// dims are packed and the remaining dims carry the source's outer strides, so the view is a
// Slice of the source's innermost axis plus a Reshape.
if (!context.is_static() && view_input_size == 1 && input.get_element_type().is_real()) {
const auto source = context.get_view_input_src_ggml_shape(input_index, 0);
const auto view = context.get_view_input_ggml_shape(input_index, 0);
const auto source_stride = context.get_view_input_src_stride(input_index, 0);
const auto view_stride = context.get_view_input_stride(input_index, 0);
const auto in_rank = input.get_partial_shape().rank();
const size_t nd = view.size();
if (nd == 4 && source.size() == nd && view_stride.size() == nd && source_stride.size() == nd &&
in_rank.is_static() && in_rank.get_length() == static_cast<int64_t>(nd) &&
view_stride[nd - 1] == source_stride[nd - 1] && source_stride[nd - 1] > 0) {
const size_t es = source_stride[nd - 1];
int j = static_cast<int>(nd) - 1;
size_t span = 1;
while (j >= 0 && view_stride[j] == span * es) {
span *= view[j];
--j;
}
// at least two packed dims replace the source's innermost dim; a one-dim span keeps the
// source rank and is handled by the generic paths below
const int shift = static_cast<int>(nd) - 2 - j;
bool matches = j >= 0 && shift >= 1;
for (int t = 0; matches && t <= j; ++t) {
matches = view[t] == source[t + shift] && view_stride[t] == source_stride[t + shift];
}
for (int t = 0; matches && t < shift; ++t) {
matches = source[t] == 1;
}
const size_t offset = context.get_view_input_offset(input_index, 0);
const size_t src_offset = context.get_view_input_src_offset(input_index, 0);
if (matches && offset >= src_offset && (offset - src_offset) % es == 0) {
const int64_t begin = static_cast<int64_t>((offset - src_offset) / es);
if (begin + static_cast<int64_t>(span) <= static_cast<int64_t>(source[nd - 1])) {
auto sliced = std::make_shared<ov::op::v8::Slice>(
input, ov::op::v0::Constant::create(ov::element::i64, {1}, {begin}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {begin + static_cast<int64_t>(span)}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {static_cast<int64_t>(nd) - 1}));
const auto view_ov_shape = context.get_view_input_ov_shape(input_index, 0);
std::vector<int64_t> pattern(nd);
size_t dynamic_dims = 0;
for (size_t i = 0; i < nd; ++i) {
if (view_ov_shape.rank().is_static() && view_ov_shape[i].is_static()) {
pattern[i] = view_ov_shape[i].get_length();
} else {
pattern[i] = -1;
++dynamic_dims;
}
}
if (dynamic_dims > 1) {
for (size_t i = 0; i < nd; ++i) {
pattern[i] = static_cast<int64_t>(view[i]);
}
}
return std::make_shared<ov::op::v1::Reshape>(
sliced, ov::op::v0::Constant::create(ov::element::i64, {nd}, pattern), false);
}
}
}
}
// A row range of a 2D weight (BERT's token type row): slice the rows of the [rows, row_size] Constant
if (view_input_size == 1 && input.get_partial_shape().rank().is_static() &&
input.get_partial_shape().rank().get_length() == 2 &&
@@ -671,14 +754,17 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
exact_chunk = i == static_cast<size_t>(split_dim) || view_ggml_shape[i] == 1 ||
view_stride_v[i] == view_src_stride_v[i];
}
// stateful execution drops the leading size-1 axis, so the input can be one rank lower than the
// ggml metadata
const auto input_rank = input.get_partial_shape().rank();
const int64_t rank_shift = input_rank.is_static() ?
static_cast<int64_t>(ndims) - input_rank.get_length() :
static_cast<int64_t>(ndims);
// num_splits is the captured source size: on a dynamic axis (e.g. the token axis of DSpark's
// result_norm, read one block position at a time) the graph is reused with other sizes, and
// each Split output would then hold several rows or the wrong one
const auto input_shape = input.get_partial_shape();
const int64_t split_axis = input_shape.rank().is_static() ?
split_dim - (int64_t) ndims + input_shape.rank().get_length() :
-1;
const bool split_axis_static = split_axis >= 0 && input_shape[split_axis].is_static() &&
input_shape[split_axis].get_length() == num_splits;
if (split_index >= 0 && split_index < num_splits && dynamic_split && exact_chunk &&
split_dim >= rank_shift) {
split_axis_static) {
// the Split keeps its input alive, so only keep a weak reference to it on the input
auto src_node = input.get_node_shared_ptr();
std::string rt_key =
@@ -689,8 +775,7 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
split_node = rt_info[rt_key].as<std::weak_ptr<ov::Node>>().lock();
}
if (split_node == nullptr) {
auto axis_const = ov::op::v0::Constant::create(
ov::element::i64, {}, {static_cast<int64_t>(split_dim) - rank_shift});
auto axis_const = ov::op::v0::Constant::create(ov::element::i64, {}, {split_axis});
split_node =
std::make_shared<ov::op::v1::Split>(input, axis_const, static_cast<size_t>(num_splits));
split_node->set_friendly_name(src_node->get_friendly_name() + "_split");
@@ -722,7 +807,8 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
// Lambda function to process a single view operation
auto process_single_view =
[](ov::Output<ov::Node> current, size_t view_offset, const std::vector<size_t> & view_stride,
[is_static = context.is_static()](
ov::Output<ov::Node> current, size_t view_offset, const std::vector<size_t> & view_stride,
const ov::Shape & view_ggml_shape, const ov::PartialShape & view_ov_shape, const std::string & view_name,
size_t view_src_offset, const std::vector<size_t> & view_src_stride, const ov::Shape & view_src_ggml_shape,
const ov::PartialShape & view_src_ov_shape, const std::string & view_src_name) -> ov::Output<ov::Node> {
@@ -847,7 +933,12 @@ ov::Output<ov::Node> process_view_input_new(const NodeContext & context, int inp
ov::op::v0::Constant::create(ov::element::i64, {1}, {1}),
ov::op::v0::Constant::create(ov::element::i64, {1}, {slice_dim - axis_shift}));
if (view_ov_shape.is_static()) {
// In a static graph a rank-preserving slice (diff_dims has one entry) already has the view
// shape. Skip the Reshape: its target bakes the static chunk size into the token axis, which
// is wrong after the output-ids GET_ROWS of the last layer (1 token in the prefill graph).
const bool rank_kept = is_static && current_rank.is_static() &&
static_cast<size_t>(current_rank.get_length()) == ndims;
if (view_ov_shape.is_static() && !rank_kept) {
auto reshaped = std::make_shared<ov::op::v1::Reshape>(
sliced,
ov::op::v0::Constant::create(ov::element::i64, {ndims}, view_ov_shape.to_shape()),
+3
View File
@@ -97,6 +97,9 @@ constexpr const char * GGML_OV_GROUPED_ROWS = "ggml_ov_grouped_rows";
// rt_info key on the stateful V cache read with `-fa off` (permute): V is [.., kv, head_size], not transposed
constexpr const char * GGML_OV_V_TOKEN_MAJOR = "ggml_ov_v_token_major";
// rt_info key on the packed SSM_SCAN output Concat [1, 1, 1, y | state]; whole-part VIEWs select one Concat input
constexpr const char * GGML_OV_SSM_SCAN_PACKED = "ggml_ov_ssm_scan_packed";
namespace op {
template <typename T> OutputVector translate_1to1_match_2_inputs(const NodeContext & context) {
num_inputs_check(context, 2, 2);
+46 -3
View File
@@ -657,6 +657,30 @@ ov::Tensor convert_ggml_input_to_ov(const std::shared_ptr<GgmlOvDecoder> & ggml_
return *sliced;
}
// KQ mask with fewer columns than the rounded KV length (see compute_llm_params): pad it with -inf
const auto & compute_params = ggml_decoder->get_compute_params();
if (!ggml_decoder->is_static() && name == compute_params.mask_name &&
compute_params.attention_size > ggml_tensor->ne[0] && ggml_is_contiguous(ggml_tensor)) {
const size_t rows = ggml_nrows(ggml_tensor);
const size_t cols = ggml_tensor->ne[0];
const size_t padded_cols = compute_params.attention_size;
ov::Shape shape = GgmlOvDecoder::get_shape(ggml_tensor);
shape.back() = padded_cols;
ov::Tensor padded(GgmlOvDecoder::get_ov_type(ggml_tensor), shape);
auto pad = [&](auto fill) {
using T = decltype(fill);
auto data = pad_input<T>((const T *) ggml_tensor->data, rows, cols, rows, padded_cols, fill);
std::memcpy(padded.data(), data.data(), padded.get_byte_size());
};
if (ggml_tensor->type == GGML_TYPE_F16) {
pad(GGML_FP32_TO_FP16(-INFINITY));
} else {
GGML_ASSERT(ggml_tensor->type == GGML_TYPE_F32);
pad(-INFINITY);
}
return padded;
}
if (ggml_decoder->is_static() &&
GgmlOvDecoder::is_kvcache(ggml_tensor, ggml_decoder->get_tensor_used_op(ggml_tensor))) {
const auto type = ggml_decoder->get_ov_type(ggml_tensor);
@@ -2192,8 +2216,21 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
old_m_params = ggml_decoder->get_model_params();
cache_hit = old_m_params.can_reuse_statically(m_params);
if (cache_hit && is_prefill && ggml_openvino_kv_slice_enabled()) {
cache_hit = entry->prefill_attention_size == c_params.attention_size &&
entry->prefill_attention_size_swa == c_params.attention_size_swa;
// A larger compiled prefill also serves a smaller attended length: the extra KV rows are masked, as with
// the power-of-two rounding. Without this, a new prompt walks up the sizes again, and since only one
// prefill model is kept, every prompt longer than one chunk recompiles. Head-major KV is bound at the
// storage size, so it keeps the exact match.
// The output count is part of the static prefill shape (more than 1 only for llama-perplexity).
const bool reuse_larger = !ggml_openvino_npu_kv_head_major_enabled();
cache_hit = entry->prefill_output_len == c_params.output_len &&
(reuse_larger ? entry->prefill_attention_size >= c_params.attention_size &&
entry->prefill_attention_size_swa >= c_params.attention_size_swa :
entry->prefill_attention_size == c_params.attention_size &&
entry->prefill_attention_size_swa == c_params.attention_size_swa);
if (cache_hit) {
c_params.attention_size = entry->prefill_attention_size;
c_params.attention_size_swa = entry->prefill_attention_size_swa;
}
prefill_size_changed = !cache_hit;
}
}
@@ -2260,6 +2297,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
entry->ptr = ggml_decoder;
entry->prefill_attention_size = c_params.attention_size;
entry->prefill_attention_size_swa = c_params.attention_size_swa;
entry->prefill_output_len = c_params.output_len;
entry->decode_attention_size = compiled.decode_attention_size;
entry->shared_key = shared_key;
entry->decode_window_requests.clear();
@@ -2274,6 +2312,10 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
r_ctx->ov_output_names_cache_prefill[key] = compiled.prefill_output_names;
decoder_end_time = conversion_end_time = compile_end_time = ggml_time_us();
GGML_LOG_DEBUG("ggml-openvino: shared compiled model HIT (static)\n");
if (prefill_size_changed) {
GGML_LOG_INFO("ggml-openvino: reuse the compiled prefill model of attention size %d\n",
c_params.attention_size);
}
} else {
auto decode_c_params = c_params;
if (ggml_openvino_kv_slice_enabled()) {
@@ -2600,6 +2642,7 @@ enum ggml_status ov_graph_compute_static(ggml_cgraph * cgraph, const std::shared
entry->ptr = ggml_decoder;
entry->prefill_attention_size = c_params.attention_size;
entry->prefill_attention_size_swa = c_params.attention_size_swa;
entry->prefill_output_len = c_params.output_len;
entry->decode_attention_size = decode_c_params.attention_size;
entry->shared_key = shared_key;
entry->decode_window_requests.clear();
@@ -3180,7 +3223,7 @@ enum ggml_status ov_graph_compute(ggml_cgraph * cgraph, ggml_backend_t backend)
GgmlOvDecoder::dump_cgraph(cgraph, filename);
}
const auto is_static = ggml_openvino_is_npu() || ggml_openvino_getenv_int("GGML_OPENVINO_FORCE_STATIC");
const auto is_static = ggml_openvino_is_static();
GGML_ASSERT(ctx->runtime_context != nullptr);
std::shared_ptr<ov_runtime_context> r_ctx = std::static_pointer_cast<ov_runtime_context>(ctx->runtime_context);
+8 -1
View File
@@ -42,6 +42,11 @@ struct graph_key {
if (GgmlOvDecoder::is_sampling_view(node)) {
input_srcs.push_back("sampling_row_" + std::string(node->name) + "#n" + std::to_string(i));
}
// The output set is part of the model (one Result per output). Enabling outputs on an existing graph,
// e.g. llama_set_embeddings_layer_inp() for a DFlash/DSpark drafter, changes only the flags.
if (node->flags & GGML_TENSOR_FLAG_OUTPUT) {
input_srcs.push_back("output_" + std::string(node->name) + "#n" + std::to_string(i));
}
}
static const bool full_key = ggml_openvino_getenv_int("GGML_OPENVINO_FULL_GRAPH_KEY") != 0;
@@ -95,9 +100,11 @@ struct decoder_runtime_ctx {
std::shared_ptr<std::mutex> mutex;
std::shared_ptr<GgmlOvDecoder> ptr;
// Attention sizes the cached prefill model was built with; decode calls overwrite the decoder's.
// Attention sizes and output count the cached prefill model was built with; decode calls overwrite the
// decoder's.
int prefill_attention_size = -1;
int prefill_attention_size_swa = -1;
int prefill_output_len = -1;
// KV window of the decode model in the compiled pair (GGML_OPENVINO_NPU_BLOB_PREBUILD).
int decode_attention_size = -1;
std::string shared_key;
+2 -2
View File
@@ -7434,8 +7434,7 @@ static bool do_ggml_backend_sycl_device_supports_op(ggml_backend_dev_t dev, cons
if (op->type == GGML_TYPE_TQ2_0 || op->type == GGML_TYPE_TQ1_0) {
return false;
}
auto res = (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16 ||
op->src[0]->type == GGML_TYPE_BF16) &&
auto res = (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16) &&
(op->src[1]->type == GGML_TYPE_I64 || op->src[1]->type == GGML_TYPE_I32);
return res;
}
@@ -7624,6 +7623,7 @@ static bool do_ggml_backend_sycl_device_supports_op(ggml_backend_dev_t dev, cons
}
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK:
return op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16;
case GGML_OP_IM2COL:
case GGML_OP_IM2COL_3D:
case GGML_OP_UPSCALE:
@@ -1043,6 +1043,8 @@ void ggml_vk_check_results_0(ggml_backend_vk_context * ctx, ggml_cgraph * cgraph
} else if (tensor->op == GGML_OP_FILL) {
const float value = ggml_get_op_params_f32(tensor, 0);
tensor_clone = ggml_fill(ggml_ctx, src_clone[0], value);
} else if (tensor->op == GGML_OP_SLEEP) {
tensor_clone = ggml_sleep(ggml_ctx, src_clone[0], ggml_get_op_params_i32(tensor, 0));
} else if (tensor->op == GGML_OP_SQR) {
tensor_clone = ggml_sqr(ggml_ctx, src_clone[0]);
} else if (tensor->op == GGML_OP_SQRT) {
@@ -189,6 +189,10 @@ struct vk_op_push_constants {
float param4;
};
struct vk_op_sleep_push_constants {
uint32_t ticks;
};
struct vk_op_fwht_push_constants {
uint32_t n_rows;
uint32_t src_offset;
+3
View File
@@ -739,6 +739,7 @@ struct vk_device_struct {
bool shader_int64;
bool buffer_device_address;
bool vulkan_memory_model;
bool shader_clock;
bool add_rms_fusion;
uint32_t partials_binding_alignment;
@@ -923,6 +924,8 @@ struct vk_device_struct {
vk_pipeline pipeline_fill_f32;
vk_pipeline pipeline_fill_f16;
vk_pipeline pipeline_sleep;
// [src/dst 0=fp32,1=fp16,2=bf16]
vk_pipeline pipeline_glu[GGML_GLU_OP_COUNT][3];
+52 -1
View File
@@ -3892,6 +3892,10 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
ggml_vk_create_pipeline(device, device->pipeline_fill_f32, "fill_f32", fill_f32_len, fill_f32_data, "main", 1, sizeof(vk_op_push_constants), {512, 1, 1}, {}, 1);
ggml_vk_create_pipeline(device, device->pipeline_fill_f16, "fill_f16", fill_f16_len, fill_f16_data, "main", 1, sizeof(vk_op_push_constants), {512, 1, 1}, {}, 1);
if (device->shader_clock) {
ggml_vk_create_pipeline(device, device->pipeline_sleep, "sleep", sleep_len, sleep_data, "main", 1, sizeof(vk_op_sleep_push_constants), {1, 1, 1}, {}, 1);
}
#define CREATE_GLU(name, op) \
ggml_vk_create_pipeline(device, device->pipeline_glu[GGML_GLU_OP_ ## op][0], #name "_f32", name ## _f32_len, name ## _f32_data, "main", 3, sizeof(vk_op_glu_push_constants), {512, 1, 1}, {}, 1, true); \
ggml_vk_create_pipeline(device, device->pipeline_glu[GGML_GLU_OP_ ## op][1], #name "_f16", name ## _f16_len, name ## _f16_data, "main", 3, sizeof(vk_op_glu_push_constants), {512, 1, 1}, {}, 1, true); \
@@ -4521,6 +4525,7 @@ vk_device ggml_vk_get_device(size_t idx) {
bool ocp_microscaling_extension = false;
bool shader_float8_extension = false;
bool workgroup_memory_explicit_layout_support = false;
bool shader_clock_support = false;
for (const auto& properties : ext_props) {
if (strcmp("VK_KHR_maintenance4", properties.extensionName) == 0) {
@@ -4571,6 +4576,9 @@ vk_device ggml_vk_get_device(size_t idx) {
} else if (strcmp(VK_EXT_SHADER_FLOAT8_EXTENSION_NAME, properties.extensionName) == 0) {
shader_float8_extension = true;
#endif
} else if (strcmp("VK_KHR_shader_clock", properties.extensionName) == 0 &&
!getenv("GGML_VK_DISABLE_SHADER_CLOCK")) {
shader_clock_support = true;
} else if (strcmp("VK_VALVE_shader_mixed_float_dot_product", properties.extensionName) == 0 &&
!getenv("GGML_VK_DISABLE_DOT2")) {
dot2_f16_support = true;
@@ -4919,6 +4927,14 @@ vk_device ggml_vk_get_device(size_t idx) {
device_extensions.push_back("VK_KHR_shader_integer_dot_product");
}
VkPhysicalDeviceShaderClockFeaturesKHR shader_clock_features {};
shader_clock_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_CLOCK_FEATURES_KHR;
if (shader_clock_support) {
last_struct->pNext = (VkBaseOutStructure *)&shader_clock_features;
last_struct = (VkBaseOutStructure *)&shader_clock_features;
device_extensions.push_back("VK_KHR_shader_clock");
}
VkPhysicalDeviceShaderMixedFloatDotProductFeaturesVALVE dot2_features {};
dot2_features.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SHADER_MIXED_FLOAT_DOT_PRODUCT_FEATURES_VALVE;
if (dot2_f16_support) {
@@ -5025,6 +5041,7 @@ vk_device ggml_vk_get_device(size_t idx) {
device->shader_int64 = device_features2.features.shaderInt64;
device->buffer_device_address = vk12_features.bufferDeviceAddress;
device->vulkan_memory_model = vk12_features.vulkanMemoryModel;
device->shader_clock = shader_clock_support && shader_clock_features.shaderDeviceClock;
if (device->subgroup_size_control) {
device->subgroup_min_size = subgroup_size_control_props.minSubgroupSize;
@@ -10235,6 +10252,8 @@ static vk_pipeline ggml_vk_op_get_pipeline(ggml_backend_vk_context * ctx, const
return ctx->device->pipeline_arange_f32;
}
return nullptr;
case GGML_OP_SLEEP:
return ctx->device->pipeline_sleep;
case GGML_OP_FILL:
if (dst->type == GGML_TYPE_F32) {
return ctx->device->pipeline_fill_f32;
@@ -11471,6 +11490,31 @@ void ggml_vk_fill(ggml_backend_vk_context * ctx, vk_context& subctx, ggml_tensor
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { dst_buf }, pc, elements);
}
static void ggml_vk_sleep(ggml_backend_vk_context * ctx, vk_context& subctx, ggml_tensor * dst) {
VK_LOG_DEBUG("ggml_vk_sleep(dst=" << dst << ", us=" << ggml_get_op_params_i32(dst, 0) << ")");
// Vulkan does not specify the period of the shader realtime clock this assumes it matches the timestamp counter.
// A driver where this does not hold could sleep too short or too long by that ratio
const float ns_per_tick = ctx->device->properties.limits.timestampPeriod;
const uint64_t ns = 1000 * (uint64_t) ggml_get_op_params_i32(dst, 0);
const uint64_t ticks = std::min<uint64_t>(ns_per_tick > 0.0f ? (uint64_t)(ns / ns_per_tick) : ns, std::numeric_limits<uint32_t>::max());
vk_op_sleep_push_constants pc = {
(uint32_t) ticks,
};
vk_pipeline pipeline = ggml_vk_op_get_pipeline(ctx, nullptr, nullptr, nullptr, dst, GGML_OP_SLEEP);
GGML_ASSERT(pipeline != nullptr);
ggml_pipeline_request_descriptor_sets(ctx, pipeline, 1);
vk_subbuffer dst_buf = ggml_vk_tensor_subbuffer(ctx, dst);
// a single invocation spins, the buffer is bound only to keep the spin from being optimized out
std::array<uint32_t, 3> elements = { 1, 1, 1 };
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline, { dst_buf }, pc, elements);
}
void ggml_vk_sin(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, ggml_tensor * dst) {
ggml_vk_op_f32(ctx, subctx, src0, nullptr, nullptr, nullptr, dst, GGML_OP_SIN, vk_op_unary_push_constants_init(src0, dst));
}
@@ -13317,6 +13361,10 @@ bool ggml_vk_build_graph(ggml_backend_vk_context * ctx, ggml_cgraph * cgraph, in
case GGML_OP_FILL:
ggml_vk_fill(ctx, compute_ctx, node);
break;
case GGML_OP_SLEEP:
ggml_vk_sleep(ctx, compute_ctx, node);
break;
case GGML_OP_SCALE:
if (ctx->fused_hc_post_gate) {
@@ -16681,8 +16729,9 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
case GGML_OP_REPEAT_BACK:
return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32;
case GGML_OP_ROPE:
return ggml_is_contiguous_rows(op) && ggml_is_contiguous_rows(op->src[0]);
return (op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16) && ggml_is_contiguous_rows(op) && ggml_is_contiguous_rows(op->src[0]);
case GGML_OP_ROPE_BACK:
return op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16;
case GGML_OP_NONE:
case GGML_OP_RESHAPE:
case GGML_OP_VIEW:
@@ -16768,6 +16817,8 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
return op->type == GGML_TYPE_F32;
case GGML_OP_FILL:
return op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16;
case GGML_OP_SLEEP:
return device->shader_clock;
case GGML_OP_SCALE:
return ggml_is_contiguous(op->src[0]) && ggml_vk_is_float_elementwise(op);
case GGML_OP_PAD:
@@ -0,0 +1,37 @@
#version 450
#extension GL_EXT_shader_realtime_clock : require
layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
// never written, it only anchors the spin below so the compiler cannot drop it
layout (binding = 0) writeonly buffer D {uint data_d[];};
layout (push_constant) uniform parameter {
uint ticks;
} p;
// bounds the spin in case the clock does not advance, turning a hang into a too-short sleep
const uint MAX_ITER = 1u << 27;
void main() {
const uvec2 t0 = clockRealtime2x32EXT();
uvec2 t = t0;
for (uint iter = 0; iter < MAX_ITER; ++iter) {
t = clockRealtime2x32EXT();
const uint lo = t.x - t0.x;
const uint hi = t.y - t0.y - (t.x < t0.x ? 1u : 0u);
if (hi != 0u || lo >= p.ticks) {
break;
}
}
// the store never runs, but the compiler cannot prove it and has to keep the spin
if (t.x == 0xFFFFFFFFu && t.y == 0xFFFFFFFFu) {
data_d[0] = t.x;
}
}
@@ -1028,6 +1028,7 @@ void process_shaders() {
string_to_spv("arange_f32", "arange.comp", {{"A_TYPE", "float"}, {"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
string_to_spv("fill_f32", "fill.comp", {{"D_TYPE", "float"}, {"FLOAT_TYPE", "float"}});
string_to_spv("fill_f16", "fill.comp", {{"D_TYPE", "float16_t"}, {"FLOAT_TYPE", "float"}});
string_to_spv("sleep", "sleep.comp", {});
string_to_spv("silu_back_f32", "silu_back.comp", {{"A_TYPE", "float"}, {"B_TYPE", "float"}, {"D_TYPE", "float"}});
@@ -1272,6 +1272,7 @@ class ggml_webgpu_shader_lib {
std::unordered_map<int, webgpu_pipeline> arange_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> count_equal_pipelines; // key is src_overlap
std::unordered_map<int, webgpu_pipeline> timestep_embedding_pipelines; // key is fixed, no variants yet
std::unordered_map<int, webgpu_pipeline> pool2d_pipelines; // key is ggml_op_pool
std::unordered_map<ggml_webgpu_row_norm_pipeline_key, webgpu_pipeline, ggml_webgpu_row_norm_pipeline_key_hash>
row_norm_pipelines; // op/inplace
@@ -2067,6 +2068,34 @@ class ggml_webgpu_shader_lib {
return arange_pipelines[1];
}
webgpu_pipeline get_pool2d_pipeline(const ggml_webgpu_shader_lib_context & context) {
const int key = ggml_get_op_params_i32(context.dst, 0);
auto it = pool2d_pipelines.find(key);
if (it != pool2d_pipelines.end()) {
return it->second;
}
std::vector<std::string> defines;
std::string variant = "pool2d";
if (key == GGML_OP_POOL_MAX) {
defines.push_back("POOL_MAX");
variant += "_max";
} else {
variant += "_avg";
}
defines.push_back(std::string("WG_SIZE=") + std::to_string(context.max_wg_size));
auto processed = preprocessor.preprocess(wgsl_pool2d, defines);
auto decisions = std::make_shared<ggml_webgpu_generic_shader_decisions>();
decisions->wg_size = context.max_wg_size;
webgpu_pipeline pipeline = ggml_webgpu_create_pipeline(device, processed, variant);
pipeline.context = decisions;
pool2d_pipelines[key] = pipeline;
return pool2d_pipelines[key];
}
webgpu_pipeline get_pad_pipeline(const ggml_webgpu_shader_lib_context & context) {
ggml_webgpu_pad_pipeline_key key = {};
key.reflect = context.dst->op == GGML_OP_PAD_REFLECT_1D;
+45
View File
@@ -961,6 +961,46 @@ static webgpu_encoded_op ggml_webgpu_pad(webgpu_context & ctx, ggml_tensor * src
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_pool2d(webgpu_context & ctx, ggml_tensor * src, ggml_tensor * dst) {
ggml_webgpu_shader_lib_context shader_lib_ctx = {};
shader_lib_ctx.src0 = src;
shader_lib_ctx.dst = dst;
shader_lib_ctx.max_wg_size = ctx->global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
webgpu_pipeline pipeline = ctx->shader_lib->get_pool2d_pipeline(shader_lib_ctx);
auto * decisions = static_cast<ggml_webgpu_generic_shader_decisions *>(pipeline.context.get());
const uint32_t ne = (uint32_t) ggml_nelements(dst);
std::vector<uint32_t> params = {
ne,
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, src) / ggml_type_size(src->type)),
(uint32_t) (ggml_webgpu_tensor_misalignment(ctx, dst) / ggml_type_size(dst->type)),
(uint32_t) src->ne[0],
(uint32_t) src->ne[1],
(uint32_t) dst->ne[0],
(uint32_t) dst->ne[1],
(uint32_t) ggml_get_op_params_i32(dst, 1), // k0
(uint32_t) ggml_get_op_params_i32(dst, 2), // k1
(uint32_t) ggml_get_op_params_i32(dst, 3), // s0
(uint32_t) ggml_get_op_params_i32(dst, 4), // s1
(uint32_t) ggml_get_op_params_i32(dst, 5), // p0
(uint32_t) ggml_get_op_params_i32(dst, 6), // p1
};
std::vector<wgpu::BindGroupEntry> entries = {
ggml_webgpu_make_tensor_bind_group_entry(ctx, 0, src),
ggml_webgpu_make_tensor_bind_group_entry(ctx, 1, dst),
};
uint32_t wg_x;
uint32_t wg_y;
uint32_t total_wg = CEIL_DIV(ne, decisions->wg_size);
compute_2d_workgroups(total_wg, ctx->global_ctx->capabilities.limits.maxComputeWorkgroupsPerDimension, wg_x, wg_y);
return ggml_backend_webgpu_build(ctx, pipeline, params, entries, wg_x, wg_y);
}
static webgpu_encoded_op ggml_webgpu_solve_tri(webgpu_context & ctx,
ggml_tensor * src0,
ggml_tensor * src1,
@@ -3670,6 +3710,8 @@ static std::optional<webgpu_encoded_op> ggml_webgpu_encode(webgpu_context ctx,
}
case GGML_OP_ARANGE:
return ggml_webgpu_arange(ctx, node);
case GGML_OP_POOL_2D:
return ggml_webgpu_pool2d(ctx, src0, node);
case GGML_OP_PAD_REFLECT_1D:
case GGML_OP_PAD:
return ggml_webgpu_pad(ctx, src0, node);
@@ -5219,6 +5261,9 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
case GGML_OP_ARANGE:
supports_op = op->type == GGML_TYPE_F32 && ggml_is_contiguous(op);
break;
case GGML_OP_POOL_2D:
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32 && ggml_is_contiguous(src0);
break;
case GGML_OP_PAD_REFLECT_1D:
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32;
break;
@@ -0,0 +1,74 @@
@group(0) @binding(0)
var<storage, read_write> src: array<f32>;
@group(0) @binding(1)
var<storage, read_write> dst: array<f32>;
struct Params {
ne: u32, // total number of elements
offset_src: u32, // in elements
offset_dst: u32, // in elements
// Logical shapes
src_ne0: u32,
src_ne1: u32,
dst_ne0: u32,
dst_ne1: u32,
k0: u32,
k1: u32,
s0: u32,
s1: u32,
p0: u32,
p1: u32
}
@group(0) @binding(2)
var<uniform> params: Params;
@compute @workgroup_size(WG_SIZE)
fn main(@builtin(num_workgroups) num_wg: vec3<u32>,
@builtin(global_invocation_id) gid: vec3<u32>) {
let gid_i = gid.x + (num_wg.x * u32(WG_SIZE)) * gid.y;
if (gid_i >= params.ne) {
return;
}
let out_x = gid_i % params.dst_ne0;
let out_y = (gid_i / params.dst_ne0) % params.dst_ne1;
let out_z = (gid_i / (params.dst_ne0 * params.dst_ne1));
let x0 = i32(out_x * params.s0) - i32(params.p0);
let y0 = i32(out_y * params.s1) - i32(params.p1);
let in_x_start = max(0, x0);
let in_x_end = min(i32(params.src_ne0), x0 + i32(params.k0));
let in_y_start = max(0, y0);
let in_y_end = min(i32(params.src_ne1), y0 + i32(params.k1));
let base = params.offset_src + out_z * params.src_ne0 * params.src_ne1;
#ifdef POOL_MAX
var res: f32 = bitcast<f32>(0xff7fffffu);
#else
var res: f32 = 0.0;
#endif
for(var y = in_y_start; y < in_y_end; y++){
let row = base + u32(y) * params.src_ne0;
for(var x = in_x_start; x < in_x_end; x++){
#ifdef POOL_MAX
res = max(res, src[row + u32(x)]);
#else
res += src[row + u32(x)];
#endif
}
}
#ifndef POOL_MAX
res = res / f32(params.k0 * params.k1);
#endif
dst[params.offset_dst + gid_i] = res;
}
+28 -3
View File
@@ -1112,9 +1112,11 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
"OPT_STEP_SGD",
"GLU",
"SLEEP",
};
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT != 99");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"none",
@@ -1225,9 +1227,11 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"sgd(x)",
"glu(x)",
"sleep(x)",
};
static_assert(GGML_OP_COUNT == 99, "GGML_OP_COUNT != 99");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
@@ -4015,7 +4019,7 @@ struct ggml_tensor * ggml_set_rows(
GGML_ASSERT(b->ne[2] % c->ne[1] == 0);
GGML_ASSERT(b->ne[3] % c->ne[2] == 0);
GGML_ASSERT(c->ne[3] == 1);
GGML_ASSERT(b->type == GGML_TYPE_F32 || b->type == GGML_TYPE_F16);
GGML_ASSERT(b->type == GGML_TYPE_F32 || b->type == GGML_TYPE_F16 || b->type == GGML_TYPE_BF16);
GGML_ASSERT(c->type == GGML_TYPE_I64 || c->type == GGML_TYPE_I32);
GGML_ASSERT(ggml_is_contiguous_rows(a));
@@ -6176,6 +6180,27 @@ struct ggml_tensor * ggml_custom_inplace(
return result;
}
// ggml_sleep
// the result aliases a, the op only delays and never touches the data
struct ggml_tensor * ggml_sleep(
struct ggml_context * ctx,
struct ggml_tensor * a,
int32_t us) {
GGML_ASSERT(us >= 0);
struct ggml_tensor * result = ggml_view_tensor(ctx, a);
ggml_format_name(result, "%s (sleep)", a->name);
ggml_set_op_params_i32(result, 0, us);
result->op = GGML_OP_SLEEP;
result->src[0] = a;
return result;
}
// ggml_cross_entropy_loss
struct ggml_tensor * ggml_cross_entropy_loss(
+208 -25
View File
@@ -1400,6 +1400,91 @@ bool llm_graph_input_mem_hybrid_iswa::can_reuse(const llm_graph_params & params)
return res;
}
void llm_graph_input_mem_hybrid_iswa_rel::set_input(const llama_ubatch * ubatch) {
const auto * attn_ctx = mctx->get_attn();
// base tensors may not be allocated if there are no non-SWA attention layers
if (inp_attn->self_k_idxs && inp_attn->self_k_idxs->buffer) {
attn_ctx->get_base()->set_input_k_idxs(inp_attn->self_k_idxs, ubatch);
attn_ctx->get_base()->set_input_v_idxs(inp_attn->self_v_idxs, ubatch);
}
if (inp_attn->self_attn_mask.raw && inp_attn->self_attn_mask.raw->buffer) {
attn_ctx->get_base()->set_input_attn_mask(inp_attn->self_attn_mask, ubatch, cparams.causal_attn);
}
if (self_rel_pos->buffer) {
attn_ctx->get_base()->set_input_relative_pos(self_rel_pos, ubatch, extent);
}
// swa tensors may not be allocated if there are no SWA attention layers
if (inp_attn->self_k_idxs_swa && inp_attn->self_k_idxs_swa->buffer) {
attn_ctx->get_swa()->set_input_k_idxs(inp_attn->self_k_idxs_swa, ubatch);
attn_ctx->get_swa()->set_input_v_idxs(inp_attn->self_v_idxs_swa, ubatch);
}
if (inp_attn->self_attn_mask_swa.raw && inp_attn->self_attn_mask_swa.raw->buffer) {
attn_ctx->get_swa()->set_input_attn_mask(inp_attn->self_attn_mask_swa, ubatch, cparams.causal_attn);
}
if (self_rel_pos_swa->buffer) {
attn_ctx->get_swa()->set_input_relative_pos(self_rel_pos_swa, ubatch, extent_swa);
}
const int64_t n_rs = mctx->get_recr()->get_n_rs();
if (inp_rs->s_copy) {
GGML_ASSERT(ggml_backend_buffer_is_host(inp_rs->s_copy->buffer));
int32_t * data = (int32_t *) inp_rs->s_copy->data;
// assuming copy destinations ALWAYS happen ONLY on the cells between head and head+n
for (uint32_t i = 0; i < n_rs; ++i) {
data[i] = mctx->get_recr()->s_copy(i);
}
}
}
bool llm_graph_input_mem_hybrid_iswa_rel::can_reuse(const llm_graph_params & params) {
const auto * mctx = static_cast<const llama_memory_hybrid_iswa_context *>(params.mctx);
this->mctx = mctx;
bool res = true;
const auto * attn_ctx = mctx->get_attn();
// the masks are always dense, see build_inp_mem_hybrid_iswa_rel()
// base tensors may not be allocated if there are no non-SWA attention layers
if (inp_attn->self_k_idxs && inp_attn->self_k_idxs->buffer) {
res &= inp_attn->self_k_idxs->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask.raw, attn_ctx->get_base(), params.ubatch, params.cparams);
res &= self_rel_pos->ne[0] == attn_ctx->get_base()->get_n_kv();
res &= self_rel_pos->ne[1] == params.ubatch.n_tokens;
// swa tensors may not be allocated if there are no SWA attention layers
if (inp_attn->self_k_idxs_swa && inp_attn->self_k_idxs_swa->buffer) {
res &= inp_attn->self_k_idxs_swa->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask_swa.raw, attn_ctx->get_swa(), params.ubatch, params.cparams);
res &= self_rel_pos_swa->ne[0] == attn_ctx->get_swa()->get_n_kv();
res &= self_rel_pos_swa->ne[1] == params.ubatch.n_tokens;
res &= inp_rs->s_copy->ne[0] == mctx->get_recr()->get_n_rs();
res &= inp_rs->s_copy_main->ne[0] == params.ubatch.n_seqs;
res &= inp_rs->head == mctx->get_recr()->get_head();
res &= inp_rs->rs_z == mctx->get_recr()->get_rs_z();
return res;
}
void llm_graph_input_sampling::set_input(const llama_ubatch * ubatch) {
// set the inputs only for the active samplers in the current ubatch
std::unordered_set<llama_seq_id> active_samplers;
@@ -3113,7 +3198,7 @@ ggml_tensor * llm_graph_context::build_attn_mha(
ggml_tensor * cur;
const bool use_flash_attn = cparams.flash_attn && kq_b == nullptr && (!sinks || ggml_is_vector(sinks));
const bool use_flash_attn = cparams.flash_attn && kq_b == nullptr;
GGML_ASSERT(use_flash_attn || kv_rows == nullptr);
if (use_flash_attn) {
GGML_ASSERT(kq_b == nullptr && "Flash attention does not support KQ bias yet");
@@ -3196,16 +3281,8 @@ ggml_tensor * llm_graph_context::build_attn_mha(
cb(kq, "kq_plus_kq_b", il);
}
if (sinks && !ggml_is_vector(sinks)) {
// Per-query sinks are extra logits with zero values.
const int64_t n_kv = kq->ne[0];
kq = ggml_add(ctx0, ggml_scale(ctx0, kq, kq_scale), ggml_cast(ctx0, kq_mask, GGML_TYPE_F32));
kq = ggml_soft_max(ctx0, ggml_concat(ctx0, kq, sinks, 0));
kq = ggml_cont(ctx0, ggml_view_4d(ctx0, kq, n_kv, kq->ne[1], kq->ne[2], kq->ne[3], kq->nb[1], kq->nb[2], kq->nb[3], 0));
} else {
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);
ggml_soft_max_add_sinks(kq, sinks);
}
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, kq_scale, hparams.f_max_alibi_bias);
ggml_soft_max_add_sinks(kq, sinks);
cb(kq, "kq_soft_max", il);
if (!v_trans) {
@@ -3687,7 +3764,9 @@ ggml_tensor * llm_graph_context::build_attn(
return cur;
}
ggml_tensor * llm_graph_context::build_attn(
// the iswa build_attn with the KQ mask and kv_rows given by the caller, which has to keep them in sync
static ggml_tensor * build_attn_iswa_impl(
const llm_graph_context & gctx,
llm_graph_input_attn_kv_iswa * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
@@ -3696,11 +3775,18 @@ ggml_tensor * llm_graph_context::build_attn(
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * kq_mask,
ggml_tensor * kv_rows,
ggml_tensor * sinks,
ggml_tensor * v_mla,
float kq_scale,
int il,
ggml_tensor * kq_mask) const {
int il) {
ggml_context * ctx0 = gctx.ctx0;
ggml_cgraph * gf = gctx.gf;
const auto & hparams = gctx.hparams;
const auto & cparams = gctx.cparams;
const bool is_swa = hparams.is_swa(il);
auto * k_rot = is_swa ? inp->self_k_rot_swa[il] : inp->self_k_rot[il];
@@ -3738,7 +3824,7 @@ ggml_tensor * llm_graph_context::build_attn(
// whole seq fits into batch in training mode
const bool use_kv_cur = cparams.training && k_cur && v_cur;
if (use_kv_cur) {
GGML_ASSERT(mctx_cur->get_n_kv() == n_tokens);
GGML_ASSERT(mctx_cur->get_n_kv() == gctx.n_tokens);
}
const bool store_kv = !use_kv_cur || hparams.n_layer_kv_from_start >= 0;
@@ -3756,25 +3842,20 @@ ggml_tensor * llm_graph_context::build_attn(
ggml_build_forward_expand(gf, mctx_cur->cpy_v(ctx0, v_cur, v_idxs, il));
}
if (!kq_mask) {
kq_mask = is_swa ? inp->self_attn_mask_swa.raw : inp->self_attn_mask.raw;
}
const auto & kv_rows = is_swa ? inp->self_attn_mask_swa.rows : inp->self_attn_mask.rows;
ggml_tensor * q = q_cur;
ggml_tensor * k = use_kv_cur ? k_cur : mctx_cur->get_k(ctx0, il);
ggml_tensor * v = use_kv_cur ? v_cur : mctx_cur->get_v(ctx0, il);
ggml_tensor * cur = build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il);
ggml_tensor * cur = gctx.build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il);
cur = build_attn_scale_kv_output(ctx0, mctx_cur, cur, il);
cb(cur, "kqv_out", il);
gctx.cb(cur, "kqv_out", il);
if (v_rot) {
cur = llama_mul_mat_hadamard(ctx0, cur, v_rot);
}
if (wo) {
cur = build_lora_mm(wo, cur, wo_s);
cur = gctx.build_lora_mm(wo, cur, wo_s);
}
if (wo_b) {
@@ -3788,6 +3869,27 @@ ggml_tensor * llm_graph_context::build_attn(
return cur;
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_attn_kv_iswa * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur,
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * sinks,
ggml_tensor * v_mla,
float kq_scale,
int il) const {
const bool is_swa = hparams.is_swa(il);
const auto & kq_mask = is_swa ? inp->self_attn_mask_swa.raw : inp->self_attn_mask.raw;
const auto & kv_rows = is_swa ? inp->self_attn_mask_swa.rows : inp->self_attn_mask.rows;
return build_attn_iswa_impl(*this, inp, wo, wo_b, wo_s, q_cur, k_cur, v_cur, kq_b, kq_mask, kv_rows, sinks, v_mla, kq_scale, il);
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_attn_k_iswa * inp,
ggml_tensor * wo,
@@ -4306,6 +4408,43 @@ llm_graph_input_mem_hybrid_iswa * llm_graph_context::build_inp_mem_hybrid_iswa()
auto inp_attn = std::make_unique<llm_graph_input_attn_kv_iswa>(hparams, cparams, attn_ctx);
{
inp_attn->self_k_idxs = attn_ctx->get_base()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs = attn_ctx->get_base()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask = build_attn_inp_attn_mask(ctx0, attn_ctx->get_base(), ubatch, cparams);
}
{
inp_attn->self_k_idxs_swa = attn_ctx->get_swa()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs_swa = attn_ctx->get_swa()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask_swa = build_attn_inp_attn_mask(ctx0, attn_ctx->get_swa(), ubatch, cparams);
}
inp_attn->self_k_rot = attn_ctx->get_base()->get_k_rot();
inp_attn->self_v_rot = attn_ctx->get_base()->get_v_rot();
inp_attn->self_k_rot_swa = attn_ctx->get_swa()->get_k_rot();
inp_attn->self_v_rot_swa = attn_ctx->get_swa()->get_v_rot();
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur);
return (llm_graph_input_mem_hybrid_iswa *) res->add_input(std::move(inp));
}
llm_graph_input_mem_hybrid_iswa_rel * llm_graph_context::build_inp_mem_hybrid_iswa_rel(
uint32_t extent,
uint32_t extent_swa) const {
const auto * mctx_cur = static_cast<const llama_memory_hybrid_iswa_context *>(mctx);
auto inp_rs = build_rs_inp_impl(ctx0, ubatch, mctx_cur->get_recr());
const auto * attn_ctx = mctx_cur->get_attn();
auto inp_attn = std::make_unique<llm_graph_input_attn_kv_iswa>(hparams, cparams, attn_ctx);
// dense masks: the bias is gathered by the distance to each cell of the K/V view
{
inp_attn->self_k_idxs = attn_ctx->get_base()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs = attn_ctx->get_base()->build_input_v_idxs(ctx0, ubatch);
@@ -4326,9 +4465,53 @@ llm_graph_input_mem_hybrid_iswa * llm_graph_context::build_inp_mem_hybrid_iswa()
inp_attn->self_k_rot_swa = attn_ctx->get_swa()->get_k_rot();
inp_attn->self_v_rot_swa = attn_ctx->get_swa()->get_v_rot();
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur);
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa_rel>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur, extent, extent_swa);
return (llm_graph_input_mem_hybrid_iswa *) res->add_input(std::move(inp));
inp->self_rel_pos = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_base()->get_n_kv(), ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos);
ggml_set_name(inp->self_rel_pos, "attn_inp_rel_pos");
inp->self_rel_pos_swa = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_swa()->get_n_kv(), ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos_swa);
ggml_set_name(inp->self_rel_pos_swa, "attn_inp_rel_pos_swa");
return (llm_graph_input_mem_hybrid_iswa_rel *) res->add_input(std::move(inp));
}
ggml_tensor * llm_graph_context::build_attn(
llm_graph_input_mem_hybrid_iswa_rel * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur,
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * sinks,
float kq_scale,
int il) const {
const bool is_swa = hparams.is_swa(il);
auto * inp_attn = inp->get_attn();
const auto & attn_mask = is_swa ? inp_attn->self_attn_mask_swa : inp_attn->self_attn_mask;
// TODO: support kv_rows - fill the distances over the kv_rows cell lists and pass the rows to build_attn_mha,
// needs a kv_rows flash attention with a per-head mask (CUDA has none)
GGML_ASSERT(attn_mask.rows == nullptr);
// a mask per head, in the dense layout of the input mask, which flash attention takes in F16
ggml_tensor * kq_mask = attn_mask.raw;
if (kq_mask->type != GGML_TYPE_F32) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F32);
}
kq_mask = ggml_add(ctx0, kq_b, kq_mask);
if (cparams.flash_attn) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F16);
}
cb(kq_mask, "kq_mask_rel_pos", il);
return build_attn_iswa_impl(*this, inp_attn, wo, wo_b, wo_s, q_cur, k_cur, v_cur, nullptr, kq_mask, nullptr, sinks, nullptr, kq_scale, il);
}
void llm_graph_context::build_dense_out(
+62 -2
View File
@@ -774,6 +774,48 @@ public:
const llama_memory_hybrid_iswa_context * mctx;
};
// hybrid iswa memory with a per-head attention bias that depends on the distance from the query to the cell (Inkling)
// the KQ masks are always dense (rows == nullptr) and are filled from the same cells as the distances, so that the bias
// gathered by distance lines up with the mask
class llm_graph_input_mem_hybrid_iswa_rel : public llm_graph_input_i {
public:
llm_graph_input_mem_hybrid_iswa_rel(
const llama_cparams & cparams,
std::unique_ptr<llm_graph_input_attn_kv_iswa> inp_attn,
std::unique_ptr<llm_graph_input_rs> inp_rs,
const llama_memory_hybrid_iswa_context * mctx,
uint32_t extent,
uint32_t extent_swa) :
inp_attn(std::move(inp_attn)),
inp_rs(std::move(inp_rs)),
extent(extent),
extent_swa(extent_swa),
cparams(cparams),
mctx(mctx) { }
virtual ~llm_graph_input_mem_hybrid_iswa_rel() = default;
void set_input(const llama_ubatch * ubatch) override;
bool can_reuse(const llm_graph_params & params) override;
std::unique_ptr<llm_graph_input_attn_kv_iswa> inp_attn;
std::unique_ptr<llm_graph_input_rs> inp_rs;
llm_graph_input_attn_kv_iswa * get_attn() const { return inp_attn.get(); }
llm_graph_input_rs * get_recr() const { return inp_rs.get(); }
// distance from the token to the cell, extent for the empty, future and more distant cells
ggml_tensor * self_rel_pos = nullptr; // I32 [n_kv, n_batch]
ggml_tensor * self_rel_pos_swa = nullptr; // I32 [n_kv_swa, n_batch]
const uint32_t extent;
const uint32_t extent_swa;
const llama_cparams cparams;
const llama_memory_hybrid_iswa_context * mctx;
};
class llm_graph_input_sampling : public llm_graph_input_i {
public:
llm_graph_input_sampling(std::map<llama_seq_id, llama_sampler *> samplers) :
@@ -1401,8 +1443,7 @@ struct llm_graph_context {
ggml_tensor * sinks, // [n_head_q]
ggml_tensor * v_mla, // [n_embd_head_v_mla, n_embd_head_v, n_head_v]
float kq_scale,
int il,
ggml_tensor * kq_mask = nullptr) const;
int il) const;
llm_graph_input_attn_k_iswa * build_attn_inp_k_iswa() const;
@@ -1495,6 +1536,25 @@ struct llm_graph_context {
llm_graph_input_mem_hybrid_iswa * build_inp_mem_hybrid_iswa() const;
// extent, extent_swa: the distance that the bias tables of the full and the SWA layers stop at
llm_graph_input_mem_hybrid_iswa_rel * build_inp_mem_hybrid_iswa_rel(
uint32_t extent,
uint32_t extent_swa) const;
// kq_b is added to the KQ mask, after the KQ scale
ggml_tensor * build_attn(
llm_graph_input_mem_hybrid_iswa_rel * inp,
ggml_tensor * wo,
ggml_tensor * wo_b,
ggml_tensor * wo_s,
ggml_tensor * q_cur, // [n_embd_head_q, n_head_q, n_tokens]
ggml_tensor * k_cur, // [n_embd_head_k, n_head_k, n_tokens]
ggml_tensor * v_cur, // [n_embd_head_v, n_head_v, n_tokens]
ggml_tensor * kq_b, // [n_kv, n_tokens/n_stream, n_head_q, n_stream]
ggml_tensor * sinks, // [n_head_q]
float kq_scale,
int il) const;
//
// pooling
//
+11 -2
View File
@@ -1,5 +1,7 @@
#include "models.h"
#include <cinttypes>
// HRM-Text: alternating low/high transformer stacks over the same token stream.
// Reference: HrmTextModel in transformers, DFM Mimir 1B.
@@ -19,8 +21,15 @@ void llama_model_hrm_text::load_arch_hparams(llama_model_loader & ml) {
GGML_ASSERT(hparams.n_hrm_l_cycles > 0);
// the GGUF block count is the expanded cache-slot count
const uint32_t n_slot = hparams.n_hrm_layers_per_stack * hparams.n_hrm_h_cycles * (hparams.n_hrm_l_cycles + 1);
GGML_ASSERT(hparams.n_layer() == n_slot);
// 64-bit: a 32-bit product can wrap and match n_layer(), then load_arch_tensors() indexes past model.layers
const uint64_t n_slot = (uint64_t) hparams.n_hrm_layers_per_stack * hparams.n_hrm_h_cycles * (hparams.n_hrm_l_cycles + 1);
if (hparams.n_hrm_layers_per_stack >= hparams.n_layer() || n_slot != hparams.n_layer()) {
throw std::runtime_error(format(
"hrm: cache-slot count %" PRIu64 " (layers_per_stack %u, h_cycles %u, l_cycles %u) "
"does not match block_count %u",
n_slot, hparams.n_hrm_layers_per_stack, hparams.n_hrm_h_cycles,
hparams.n_hrm_l_cycles, hparams.n_layer()));
}
switch (hparams.n_embd) {
case 1536:
+290 -196
View File
@@ -2,111 +2,157 @@
#include "../llama-memory-hybrid-iswa.h"
void llama_model_inkling2::load_arch_hparams(llama_model_loader & ml) {
ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all);
ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
ml.get_key(LLM_KV_ATTENTION_QK_CLIP, hparams.f_qk_clip);
ml.get_key(LLM_KV_ATTN_RES_BLOCK_SIZE, hparams.attn_res_block_size);
ml.get_key(LLM_KV_SHORTCONV_L_CACHE, hparams.n_shortconv_l_cache);
// relative-position bias, and the log-scaled positions of the full-attention layers
ml.get_key(LLM_KV_ATTENTION_RELATIVE_LENGTH, hparams.n_relative);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_EXTENT, hparams.n_relative_extent);
ml.get_key(LLM_KV_ATTENTION_QK_CLIP, hparams.f_qk_clip);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_CLIP, hparams.f_relative_clip);
ml.get_key(LLM_KV_ATTENTION_RELATIVE_CLIP, hparams.f_relative_clip);
ml.get_key(LLM_KV_ATTENTION_LOG_SCALE_FLOOR, hparams.n_log_scale_floor);
ml.get_key(LLM_KV_ATTENTION_LOG_SCALE_ALPHA, hparams.f_log_scale_alpha);
ml.get_key(LLM_KV_ATTN_RES_BLOCK_SIZE, hparams.attn_res_block_size);
ml.get_key(LLM_KV_SHORTCONV_L_CACHE, hparams.n_shortconv_l_cache);
ml.get_key(LLM_KV_VOCAB_SIZE, hparams.n_vocab_padded);
// the embedding and output tensors are padded past the vocabulary
ml.get_key(LLM_KV_VOCAB_SIZE, hparams.n_vocab_padded);
ml.get_key(LLM_KV_VOCAB_SIZE_UNPADDED, hparams.n_vocab_unpadded);
ml.get_key(LLM_KV_ACTIVATION_SITU_BETA, hparams.situ_beta);
// MoE
ml.get_key_or_arr(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp_arr, hparams.n_layer_all);
ml.get_key(LLM_KV_ACTIVATION_SITU_BETA, hparams.situ_beta);
ml.get_key(LLM_KV_ACTIVATION_SITU_LINEAR_BETA, hparams.situ_linear_beta);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale);
ml.get_key(LLM_KV_FINAL_LOGIT_SOFTCAPPING, hparams.f_final_logit_softcapping);
ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale, false);
hparams.f_final_logit_softcapping = 0.0f;
ml.get_key(LLM_KV_FINAL_LOGIT_SOFTCAPPING, hparams.f_final_logit_softcapping, false);
// sliding window
ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, hparams.n_swa);
ml.get_arr(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, hparams.is_swa_impl);
uint32_t n_pattern = 0;
ml.get_arr_n(LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN, n_pattern);
if (n_pattern != hparams.n_layer() || !hparams.n_relative || !hparams.n_relative_extent || !hparams.n_swa ||
!hparams.attn_res_block_size || !hparams.n_log_scale_floor || hparams.n_shortconv_l_cache < 2 ||
hparams.n_shortconv_l_cache > 64 || hparams.n_relative_extent > INT32_MAX || hparams.n_swa > INT32_MAX ||
!hparams.n_vocab_unpadded || hparams.n_vocab_unpadded > hparams.n_vocab_padded) {
throw std::runtime_error("inkling2: invalid dimensions, vocabulary, or layer pattern");
}
for (float value : {hparams.f_norm_rms_eps, hparams.f_qk_clip, hparams.f_relative_clip, hparams.situ_beta,
hparams.situ_linear_beta, hparams.expert_weights_scale, hparams.f_final_logit_softcapping}) {
for (float value : {hparams.f_norm_rms_eps, hparams.f_qk_clip, hparams.f_relative_clip, hparams.situ_beta, hparams.situ_linear_beta}) {
if (!std::isfinite(value) || value <= 0.0f) {
throw std::runtime_error("inkling2: normalization, clipping and scaling parameters must be finite and positive");
}
}
if (!std::isfinite(hparams.f_log_scale_alpha) || hparams.f_log_scale_alpha < 0.0f) {
throw std::runtime_error("inkling2: invalid logarithmic scale");
}
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
hparams.expert_router_f32 = true;
hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
hparams.expert_router_f32 = true;
hparams.expert_weights_norm_eps = 1e-20f;
// no RoPE, the relative-position bias carries the positions
hparams.rope_pattern.fill(false);
uint64_t max_channels = 0;
// the convolution state keeps K, V, the attention output and the FFN output of the widest layer
uint64_t n_channels_max = 0;
for (uint32_t il = 0; il < hparams.n_layer(); ++il) {
if (!hparams.n_head_kv(il) || hparams.n_head(il) % hparams.n_head_kv(il) ||
hparams.n_embd_head_k(il) != hparams.n_embd_head_v(il) || !hparams.n_expert_used(il) ||
hparams.n_expert_used(il) > hparams.n_expert || !hparams.n_ff_exp(il)) {
throw std::runtime_error("inkling2: invalid attention or expert dimensions");
}
max_channels = std::max<uint64_t>(max_channels, 2ull * hparams.n_embd_k_gqa(il) + 2ull * hparams.n_embd);
n_channels_max = std::max<uint64_t>(n_channels_max, 2ull * hparams.n_embd_k_gqa(il) + 2ull * hparams.n_embd);
}
if (max_channels * (hparams.n_shortconv_l_cache - 1) > UINT32_MAX) {
if (n_channels_max * (hparams.n_shortconv_l_cache - 1) > UINT32_MAX) {
throw std::runtime_error("inkling2: convolution state is too large");
}
hparams.n_sconv_channels = max_channels;
hparams.n_sconv_channels = n_channels_max;
type = LLM_TYPE_UNKNOWN;
}
void llama_model_inkling2::load_arch_tensors(llama_model_loader &) {
LLAMA_LOAD_LOCALS;
if (hparams.n_vocab_unpadded != vocab.n_tokens()) {
throw std::runtime_error("inkling2: tokenizer size must equal the unpadded vocabulary size");
}
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, hparams.n_vocab_padded}, 0);
const int64_t n_vocab_padded = hparams.n_vocab_padded;
const int64_t n_rel = hparams.n_relative;
const int64_t n_conv = hparams.n_shortconv_l_cache;
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab_padded}, 0);
tok_norm = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD_NORM, "weight", 0), {n_embd}, 0);
output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, hparams.n_vocab_padded}, 0);
output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
// output
output = create_tensor(tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab_padded}, 0);
output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
output_res_query = create_tensor(tn(LLM_TENSOR_OUTPUT_RES_QUERY, "weight"), {n_embd}, 0);
for (int i = 0; i < n_layer; ++i) {
auto & l = layers[i];
const int64_t nh = hparams.n_head(i);
const int64_t nk = hparams.n_head_kv(i);
const int64_t hd = hparams.n_embd_head_k(i);
const int64_t nr = hparams.n_relative;
const int64_t kernel = hparams.n_shortconv_l_cache;
const int64_t extent = hparams.is_swa(i) ? hparams.n_swa : hparams.n_relative_extent;
l.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), {n_embd, nh*hd}, 0);
l.wk = create_tensor(tn(LLM_TENSOR_ATTN_K, "weight", i), {n_embd, nk*hd}, 0);
l.wv = create_tensor(tn(LLM_TENSOR_ATTN_V, "weight", i), {n_embd, nk*hd}, 0);
l.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {nh*hd, n_embd}, 0);
l.wqkv_gate = create_tensor(tn(LLM_TENSOR_ATTN_GATE, "weight", i), {n_embd, nh*hd}, 0);
l.attn_rel = create_tensor(tn(LLM_TENSOR_ATTN_REL, "weight", i), {n_embd, nh*nr}, 0);
l.attn_rel_bias = create_tensor(tn(LLM_TENSOR_ATTN_REL_BIAS, "weight", i), {nr, extent}, 0);
l.attn_q_clip = create_tensor(tn(LLM_TENSOR_ATTN_Q_CLIP, "weight", i), {hd, nh}, 0);
l.attn_rel_clip = create_tensor(tn(LLM_TENSOR_ATTN_REL_CLIP, "weight", i), {nr, nh}, 0);
l.attn_sink_proj = create_tensor(tn(LLM_TENSOR_ATTN_SINK_PROJ, "weight", i), {n_embd, nh}, 0);
l.attn_k_conv = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV, "weight", i), {kernel, nk*hd}, 0);
l.attn_v_conv = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV, "weight", i), {kernel, nk*hd}, 0);
l.attn_post_conv = create_tensor(tn(LLM_TENSOR_ATTN_POST_CONV, "weight", i), {kernel, n_embd}, 0);
l.ffn_post_conv = create_tensor(tn(LLM_TENSOR_FFN_POST_CONV, "weight", i), {kernel, n_embd}, 0);
auto & layer = layers[i];
const int64_t n_head = hparams.n_head(i);
const int64_t n_head_kv = hparams.n_head_kv(i);
const int64_t n_embd_head = hparams.n_embd_head_k(i);
const int64_t n_ff_exp = hparams.n_ff_exp(i);
const int64_t n_extent = hparams.is_swa(i) ? hparams.n_swa : hparams.n_relative_extent;
// attention
layer.wq = create_tensor(tn(LLM_TENSOR_ATTN_Q, "weight", i), {n_embd, n_embd_head*n_head}, 0);
layer.wk = create_tensor(tn(LLM_TENSOR_ATTN_K, "weight", i), {n_embd, n_embd_head*n_head_kv}, 0);
layer.wv = create_tensor(tn(LLM_TENSOR_ATTN_V, "weight", i), {n_embd, n_embd_head*n_head_kv}, 0);
layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_embd_head*n_head, n_embd}, 0);
layer.wqkv_gate = create_tensor(tn(LLM_TENSOR_ATTN_GATE, "weight", i), {n_embd, n_embd_head*n_head}, 0);
layer.attn_q_clip = create_tensor(tn(LLM_TENSOR_ATTN_Q_CLIP, "weight", i), {n_embd_head, n_head}, 0);
layer.attn_sink_proj = create_tensor(tn(LLM_TENSOR_ATTN_SINK_PROJ, "weight", i), {n_embd, n_head}, 0);
// relative-position bias
layer.attn_rel = create_tensor(tn(LLM_TENSOR_ATTN_REL, "weight", i), {n_embd, n_rel*n_head}, 0);
layer.attn_rel_bias = create_tensor(tn(LLM_TENSOR_ATTN_REL_BIAS, "weight", i), {n_rel, n_extent}, 0);
layer.attn_rel_clip = create_tensor(tn(LLM_TENSOR_ATTN_REL_CLIP, "weight", i), {n_rel, n_head}, 0);
// short convolutions
layer.attn_k_conv = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV, "weight", i), {n_conv, n_embd_head*n_head_kv}, 0);
layer.attn_v_conv = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV, "weight", i), {n_conv, n_embd_head*n_head_kv}, 0);
layer.attn_post_conv = create_tensor(tn(LLM_TENSOR_ATTN_POST_CONV, "weight", i), {n_conv, n_embd}, 0);
layer.ffn_post_conv = create_tensor(tn(LLM_TENSOR_FFN_POST_CONV, "weight", i), {n_conv, n_embd}, 0);
if (hparams.is_swa(i)) {
l.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);
} else {
l.attn_q_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_k_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_v_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_V_INP_NORM, "weight", i), {n_embd}, 0);
l.attn_k_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV_PROJ, "weight", i), {n_embd, kernel*nk}, 0);
l.attn_v_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV_PROJ, "weight", i), {n_embd, kernel*nk}, 0);
// full-attention layers: separate Q, K and V input norms, and input-dependent taps for the K and V convolutions
layer.attn_q_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_Q_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_k_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_K_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_v_inp_norm = create_tensor(tn(LLM_TENSOR_ATTN_V_INP_NORM, "weight", i), {n_embd}, 0);
layer.attn_k_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_K_CONV_PROJ, "weight", i), {n_embd, n_conv*n_head_kv}, 0);
layer.attn_v_conv_proj = create_tensor(tn(LLM_TENSOR_ATTN_V_CONV_PROJ, "weight", i), {n_embd, n_conv*n_head_kv}, 0);
}
// block attention residuals
if (i > 0) {
l.attn_res_query = create_tensor(tn(LLM_TENSOR_ATTN_RES_QUERY, "weight", i), {n_embd, hparams.is_swa(i) ? 1 : 3}, 0);
layer.attn_res_query = create_tensor(tn(LLM_TENSOR_ATTN_RES_QUERY, "weight", i), {n_embd, hparams.is_swa(i) ? 1 : 3}, 0);
}
l.ffn_res_query = create_tensor(tn(LLM_TENSOR_FFN_RES_QUERY, "weight", i), {n_embd}, 0);
l.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
l.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0);
l.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, TENSOR_NOT_REQUIRED);
l.ffn_gate_up_exps = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "weight", i), {n_embd, 2*hparams.n_ff_exp(i), n_expert}, 0);
l.ffn_down_exps = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {hparams.n_ff_exp(i), n_embd, n_expert}, 0);
layer.ffn_res_query = create_tensor(tn(LLM_TENSOR_FFN_RES_QUERY, "weight", i), {n_embd}, 0);
// MoE
layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);
layer.ffn_gate_inp = create_tensor(tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, 0);
layer.ffn_exp_probs_b = create_tensor(tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, TENSOR_NOT_REQUIRED);
layer.ffn_gate_up_exps = create_tensor(tn(LLM_TENSOR_FFN_GATE_UP_EXPS, "weight", i), {n_embd, 2*n_ff_exp, n_expert}, 0);
layer.ffn_down_exps = create_tensor(tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {n_ff_exp, n_embd, n_expert}, 0);
}
}
@@ -114,202 +160,250 @@ std::unique_ptr<llm_graph_context> llama_model_inkling2::build_arch_graph(const
return std::make_unique<graph>(params);
}
class llm_graph_input_inkling2_pos : public llm_graph_input_i {
public:
const llama_kv_cache_context * cache[2];
ggml_tensor * distance[2];
uint32_t extent[2];
bool can_reuse(const llm_graph_params & params) override {
const auto * memory = static_cast<const llama_memory_hybrid_iswa_context *>(params.mctx);
cache[0] = memory->get_attn()->get_base();
cache[1] = memory->get_attn()->get_swa();
for (int i = 0; i < 2; ++i) {
if (distance[i]->ne[0] != cache[i]->get_n_kv() || distance[i]->ne[1] != params.ubatch.n_tokens) {
return false;
}
}
return true;
}
void set_input(const llama_ubatch * ubatch) override {
for (int i = 0; i < 2; ++i) {
if (distance[i]->buffer) {
cache[i]->set_input_relative_pos(distance[i], ubatch, extent[i]);
}
}
}
};
// scale each row down to an L2 norm of at most threshold, after the optional per-channel scale
ggml_tensor * llama_model_inkling2::graph::clip(ggml_tensor * cur, ggml_tensor * scale, float threshold) {
if (scale) {
cur = ggml_mul(ctx0, cur, scale);
}
auto * norm = ggml_sqrt(ctx0, ggml_sum_rows(ctx0, ggml_sqr(ctx0, cur)));
ggml_tensor * norm = ggml_sqrt(ctx0, ggml_sum_rows(ctx0, ggml_sqr(ctx0, cur)));
norm = ggml_scale_bias(ctx0, norm, 1.0f, 1e-6f);
auto * factor = ggml_div(ctx0, ggml_fill(ctx0, norm, threshold), norm);
ggml_tensor * factor = ggml_div(ctx0, ggml_fill(ctx0, norm, threshold), norm);
return ggml_mul(ctx0, cur, ggml_clamp(ctx0, factor, 0.0f, 1.0f));
}
// attention over the depth: mix the earlier block outputs and cur with learned queries
ggml_tensor * llama_model_inkling2::graph::mix(const std::vector<ggml_tensor *> & blocks, ggml_tensor * cur, ggml_tensor * query) {
auto * states = ggml_reshape_3d(ctx0, cur, n_embd, 1, n_tokens);
for (auto * block : blocks) {
ggml_tensor * states = ggml_reshape_3d(ctx0, cur, n_embd, 1, n_tokens);
for (ggml_tensor * block : blocks) {
states = ggml_concat(ctx0, ggml_reshape_3d(ctx0, block, n_embd, 1, n_tokens), states, 1);
}
states = ggml_rms_norm(ctx0, states, 1e-6f);
auto * queries = ggml_repeat(ctx0, query, ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, n_embd, query->ne[1], n_tokens));
auto * scores = ggml_mul_mat(ctx0, states, queries);
ggml_tensor * queries = ggml_repeat(ctx0, query, ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, n_embd, query->ne[1], n_tokens));
ggml_tensor * scores = ggml_mul_mat(ctx0, states, queries);
scores = ggml_soft_max_ext(ctx0, scores, nullptr, 1.0f/sqrtf(n_embd), 0.0f);
return ggml_mul_mat(ctx0, ggml_cont(ctx0, ggml_transpose(ctx0, states)), scores);
}
ggml_tensor * llama_model_inkling2::graph::convolve(ggml_tensor * cur, ggml_tensor * weight, ggml_tensor * dynamic,
ggml_tensor * state, ggml_tensor * history, llm_graph_input_rs * inp, int64_t offset) {
const int64_t kernel = hparams.n_shortconv_l_cache;
const int64_t channels = cur->ne[0];
const int64_t nt = ubatch.n_seq_tokens;
const int64_t ns = ubatch.n_seqs;
auto * past = ggml_view_3d(ctx0, history, kernel - 1, channels, ns, history->nb[1], history->nb[2], offset*history->nb[1]);
auto * x = ggml_transpose(ctx0, ggml_reshape_3d(ctx0, cur, channels, nt, ns));
// causal short convolution with a residual; dynamic adds input-dependent taps per head
// the last n_conv - 1 inputs of each sequence live in the recurrent state at channel offset `offset`
ggml_tensor * llama_model_inkling2::graph::convolve(
ggml_tensor * cur,
ggml_tensor * weight,
ggml_tensor * dynamic,
ggml_tensor * state,
ggml_tensor * history,
llm_graph_input_rs * inp,
int64_t offset) {
const int64_t n_conv = hparams.n_shortconv_l_cache;
const int64_t n_channels = cur->ne[0];
const int64_t n_seq_tokens = ubatch.n_seq_tokens;
const int64_t n_seqs = ubatch.n_seqs;
ggml_tensor * past = ggml_view_3d(ctx0, history, n_conv - 1, n_channels, n_seqs, history->nb[1], history->nb[2], offset*history->nb[1]);
ggml_tensor * x = ggml_transpose(ctx0, ggml_reshape_3d(ctx0, cur, n_channels, n_seq_tokens, n_seqs));
x = ggml_concat(ctx0, past, x, 0);
build_conv_state_store(inp, state, x, kernel - 1, offset*(kernel - 1)*ggml_element_size(state));
auto * out = ggml_ssm_conv(ctx0, x, weight);
out = ggml_reshape_2d(ctx0, out, channels, n_tokens);
build_conv_state_store(inp, state, x, n_conv - 1, offset*(n_conv - 1)*ggml_element_size(state));
ggml_tensor * out = ggml_ssm_conv(ctx0, x, weight);
out = ggml_reshape_2d(ctx0, out, n_channels, n_tokens);
if (dynamic) {
const int64_t heads = dynamic->ne[0]/kernel;
dynamic = ggml_reshape_3d(ctx0, dynamic, kernel, heads, n_tokens);
for (int64_t lag = 0; lag < kernel; ++lag) {
auto * tap = ggml_view_3d(ctx0, dynamic, 1, heads, n_tokens, dynamic->nb[1], dynamic->nb[2], lag*dynamic->nb[0]);
auto * window = ggml_view_3d(ctx0, x, nt, channels, ns, x->nb[1], x->nb[2], (kernel - 1 - lag)*x->nb[0]);
const int64_t n_heads = dynamic->ne[0]/n_conv;
dynamic = ggml_reshape_3d(ctx0, dynamic, n_conv, n_heads, n_tokens);
for (int64_t lag = 0; lag < n_conv; ++lag) {
ggml_tensor * tap = ggml_view_3d(ctx0, dynamic, 1, n_heads, n_tokens, dynamic->nb[1], dynamic->nb[2], lag*dynamic->nb[0]);
ggml_tensor * window = ggml_view_3d(ctx0, x, n_seq_tokens, n_channels, n_seqs, x->nb[1], x->nb[2], (n_conv - 1 - lag)*x->nb[0]);
window = ggml_cont(ctx0, ggml_transpose(ctx0, window));
window = ggml_reshape_3d(ctx0, window, channels/heads, heads, n_tokens);
auto * term = ggml_reshape_2d(ctx0, ggml_mul(ctx0, window, tap), channels, n_tokens);
window = ggml_reshape_3d(ctx0, window, n_channels/n_heads, n_heads, n_tokens);
ggml_tensor * term = ggml_reshape_2d(ctx0, ggml_mul(ctx0, window, tap), n_channels, n_tokens);
out = ggml_add(ctx0, out, term);
}
}
return ggml_scale(ctx0, ggml_add(ctx0, cur, out), 1.0f/sqrtf((dynamic ? 2 : 1)*kernel + 1));
return ggml_scale(ctx0, ggml_add(ctx0, cur, out), 1.0f/sqrtf((dynamic ? 2 : 1)*n_conv + 1));
}
llama_model_inkling2::graph::graph(const llm_graph_params & params) : llm_graph_context(params) {
auto * inp = build_inp_mem_hybrid_iswa();
auto * inp = build_inp_mem_hybrid_iswa_rel(hparams.n_relative_extent, hparams.n_swa);
auto * attn = inp->get_attn();
auto * recr = inp->get_recr();
auto * positions = build_inp_pos();
auto * out_ids = build_inp_out_ids();
auto rel = std::make_unique<llm_graph_input_inkling2_pos>();
rel->cache[0] = attn->mctx->get_base();
rel->cache[1] = attn->mctx->get_swa();
rel->extent[0] = hparams.n_relative_extent;
rel->extent[1] = hparams.n_swa;
ggml_tensor * distances[2];
for (int i = 0; i < 2; ++i) {
distances[i] = rel->distance[i] = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, rel->cache[i]->get_n_kv(), n_tokens);
ggml_set_input(distances[i]);
}
res->add_input(std::move(rel));
auto * tau = ggml_scale_bias(ctx0, ggml_cast(ctx0, positions, GGML_TYPE_F32), 1.0f, 1.0f);
ggml_tensor * inp_pos = build_inp_pos();
ggml_tensor * inp_out_ids = build_inp_out_ids();
// log-scaled positions: 1 + alpha*log(max(1, (pos + 1)/floor)), multiplies q and r of the full-attention layers
ggml_tensor * tau = ggml_scale_bias(ctx0, ggml_cast(ctx0, inp_pos, GGML_TYPE_F32), 1.0f, 1.0f);
tau = ggml_log(ctx0, ggml_clamp(ctx0, ggml_scale(ctx0, tau, 1.0f/hparams.n_log_scale_floor), 1.0f, INFINITY));
tau = ggml_reshape_3d(ctx0, ggml_scale_bias(ctx0, tau, hparams.f_log_scale_alpha, 1.0f), 1, 1, n_tokens);
auto * cur = build_inp_embd(model.tok_embd);
ggml_tensor * cur = build_inp_embd(model.tok_embd);
// important: do not normalize raw embeddings input (i.e. encoded image embeddings)
if (ubatch.token) {
cur = build_norm(cur, model.tok_norm, nullptr, LLM_NORM_RMS, -1);
}
std::vector<ggml_tensor *> blocks;
for (int il = 0; il < n_layer; ++il) {
const auto & l = model.layers[il];
const bool swa = hparams.is_swa(il);
const int64_t nh = hparams.n_head(il);
const int64_t nk = hparams.n_head_kv(il);
const int64_t hd = hparams.n_embd_head_k(il);
const int64_t kv = nk*hd;
auto * state = recr->mctx->get_r_l(il);
auto * history = build_rs(recr, state, hparams.n_embd_r(), ubatch.n_seqs);
const auto & layer = model.layers[il];
const bool is_swa = hparams.is_swa(il);
const int64_t n_head = hparams.n_head(il);
const int64_t n_head_kv = hparams.n_head_kv(il);
const int64_t n_embd_head = hparams.n_embd_head_k(il);
const int64_t n_embd_k_gqa = n_embd_head*n_head_kv;
ggml_tensor * state = recr->mctx->get_r_l(il);
ggml_tensor * history = build_rs(recr, state, hparams.n_embd_r(), ubatch.n_seqs);
history = ggml_reshape_3d(ctx0, history, hparams.n_shortconv_l_cache - 1, hparams.n_sconv_channels, ubatch.n_seqs);
auto * residual = cur;
ggml_tensor * inpSA = cur;
// attention inputs: SWA layers mix one, full-attention layers one each for Q, K and V
ggml_tensor * inputs[3] = {cur, cur, cur};
if (l.attn_res_query) {
auto * mixed = mix(blocks, cur, l.attn_res_query);
if (layer.attn_res_query) {
ggml_tensor * mixed = mix(blocks, cur, layer.attn_res_query);
for (int j = 0; j < 3; ++j) {
inputs[j] = ggml_cont(ctx0, ggml_view_2d(ctx0, mixed, n_embd, n_tokens, mixed->nb[2], (swa ? 0 : j)*mixed->nb[1]));
inputs[j] = ggml_cont(ctx0, ggml_view_2d(ctx0, mixed, n_embd, n_tokens, mixed->nb[2], (is_swa ? 0 : j)*mixed->nb[1]));
}
}
if (swa) {
inputs[0] = build_norm(inputs[0], l.attn_norm, nullptr, LLM_NORM_RMS, il);
if (is_swa) {
inputs[0] = build_norm(inputs[0], layer.attn_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = inputs[2] = inputs[0];
} else {
inputs[0] = build_norm(inputs[0], l.attn_q_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = build_norm(inputs[1], l.attn_k_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[2] = build_norm(inputs[2], l.attn_v_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[0] = build_norm(inputs[0], layer.attn_q_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[1] = build_norm(inputs[1], layer.attn_k_inp_norm, nullptr, LLM_NORM_RMS, il);
inputs[2] = build_norm(inputs[2], layer.attn_v_inp_norm, nullptr, LLM_NORM_RMS, il);
}
auto * q = build_lora_mm(l.wq, inputs[0]);
auto * k = convolve(build_lora_mm(l.wk, inputs[1]), l.attn_k_conv,
swa ? nullptr : build_lora_mm(l.attn_k_conv_proj, inputs[1]), state, history, recr, 0);
auto * v = convolve(build_lora_mm(l.wv, inputs[2]), l.attn_v_conv,
swa ? nullptr : build_lora_mm(l.attn_v_conv_proj, inputs[2]), state, history, recr, kv);
auto * r = build_lora_mm(l.attn_rel, inputs[0]);
q = clip(ggml_reshape_3d(ctx0, q, hd, nh, n_tokens), l.attn_q_clip, hparams.f_qk_clip);
k = clip(ggml_reshape_3d(ctx0, k, hd, nk, n_tokens), nullptr, hparams.f_qk_clip);
v = ggml_rms_norm(ctx0, ggml_reshape_3d(ctx0, v, hd, nk, n_tokens), hparams.f_norm_rms_eps);
r = clip(ggml_reshape_3d(ctx0, r, hparams.n_relative, nh, n_tokens), l.attn_rel_clip, hparams.f_relative_clip);
if (!swa) {
q = ggml_mul(ctx0, q, tau);
r = ggml_mul(ctx0, r, tau);
// self-attention
{
ggml_tensor * Qcur = build_lora_mm(layer.wq, inputs[0]);
ggml_tensor * Kcur = convolve(build_lora_mm(layer.wk, inputs[1]), layer.attn_k_conv,
is_swa ? nullptr : build_lora_mm(layer.attn_k_conv_proj, inputs[1]), state, history, recr, 0);
ggml_tensor * Vcur = convolve(build_lora_mm(layer.wv, inputs[2]), layer.attn_v_conv,
is_swa ? nullptr : build_lora_mm(layer.attn_v_conv_proj, inputs[2]), state, history, recr, n_embd_k_gqa);
ggml_tensor * Rcur = build_lora_mm(layer.attn_rel, inputs[0]);
Qcur = clip(ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head, n_tokens), layer.attn_q_clip, hparams.f_qk_clip);
Kcur = clip(ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens), nullptr, hparams.f_qk_clip);
Vcur = ggml_rms_norm(ctx0, ggml_reshape_3d(ctx0, Vcur, n_embd_head, n_head_kv, n_tokens), hparams.f_norm_rms_eps);
Rcur = clip(ggml_reshape_3d(ctx0, Rcur, hparams.n_relative, n_head, n_tokens), layer.attn_rel_clip, hparams.f_relative_clip);
if (!is_swa) {
Qcur = ggml_mul(ctx0, Qcur, tau);
Rcur = ggml_mul(ctx0, Rcur, tau);
}
cb(Qcur, "Qcur", il);
cb(Kcur, "Kcur", il);
cb(Vcur, "Vcur", il);
cb(Rcur, "Rcur", il);
ggml_tensor * rel_pos = is_swa ? inp->self_rel_pos_swa : inp->self_rel_pos;
const int64_t n_stream = (is_swa ? attn->self_attn_mask_swa.raw : attn->self_attn_mask.raw)->ne[3];
const int64_t n_kv = rel_pos->ne[0];
ggml_tensor * sinks = build_lora_mm(layer.attn_sink_proj, inputs[0]);
// project r on the bias table first, then gather by distance: only the gathered bias is n_kv x n_tokens x n_head
ggml_tensor * table = ggml_mul_mat(ctx0, ggml_pad(ctx0, layer.attn_rel_bias, 0, 1, 0, 0), Rcur);
// the sink logit t is per query: exp(s - t)/(1 + sum(exp(s - t))) = exp(s)/(exp(t) + sum(exp(s))), with a zero sink
table = ggml_sub(ctx0, table, ggml_reshape_3d(ctx0, sinks, 1, n_head, n_tokens));
table = ggml_cont(ctx0, ggml_permute(ctx0, table, 1, 0, 2, 3));
ggml_tensor * kq_b = ggml_get_rows(ctx0, table, rel_pos);
kq_b = ggml_reshape_4d(ctx0, kq_b, n_head, n_kv, n_tokens/n_stream, n_stream);
kq_b = ggml_cont(ctx0, ggml_permute(ctx0, kq_b, 2, 0, 1, 3));
cb(kq_b, "kq_b", il);
sinks = ggml_fill(ctx0, ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, n_head), 0.0f);
cur = build_attn(inp,
nullptr, nullptr, nullptr,
Qcur, Kcur, Vcur, kq_b, sinks, 1.0f/sqrtf(n_embd_head), il);
cur = ggml_mul(ctx0, cur, ggml_sigmoid(ctx0, build_lora_mm(layer.wqkv_gate, inputs[0])));
cur = convolve(build_lora_mm(layer.wo, cur), layer.attn_post_conv, nullptr, state, history, recr, 2*n_embd_k_gqa);
cb(cur, "attn_out", il);
}
auto * mask = swa ? attn->self_attn_mask_swa.raw : attn->self_attn_mask.raw;
const int64_t streams = mask->ne[3];
const int64_t n_kv = distances[swa]->ne[0];
auto * sinks = build_lora_mm(l.attn_sink_proj, inputs[0]);
// project r on the bias table first, then gather by distance: only the gathered bias is n_kv x n_tokens x nh
auto * table = ggml_mul_mat(ctx0, ggml_pad(ctx0, l.attn_rel_bias, 0, 1, 0, 0), r);
if (cparams.flash_attn) {
// exp(s - t)/(1 + sum(exp(s - t))) = exp(s)/(exp(t) + sum(exp(s))).
table = ggml_sub(ctx0, table, ggml_reshape_3d(ctx0, sinks, 1, nh, n_tokens));
} else {
// build_attn scales the bias along with QK; FA adds the mask after scaling QK.
table = ggml_scale(ctx0, table, sqrtf(hd));
}
table = ggml_cont(ctx0, ggml_permute(ctx0, table, 1, 0, 2, 3));
auto * bias = ggml_get_rows(ctx0, table, distances[swa]);
bias = ggml_cont(ctx0, ggml_permute(ctx0, ggml_reshape_4d(ctx0, bias, nh, n_kv, n_tokens/streams, streams), 2, 0, 1, 3));
ggml_tensor * fa_mask = nullptr;
if (cparams.flash_attn) {
fa_mask = ggml_cast(ctx0, ggml_add(ctx0, bias, ggml_cast(ctx0, mask, GGML_TYPE_F32)), GGML_TYPE_F16);
sinks = ggml_fill(ctx0, ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, nh), 0.0f);
bias = nullptr;
} else {
sinks = ggml_reshape_4d(ctx0, ggml_cont(ctx0, ggml_permute(ctx0,
ggml_reshape_3d(ctx0, sinks, nh, n_tokens/streams, streams), 1, 0, 2, 3)), 1, n_tokens/streams, nh, streams);
}
cur = build_attn(attn, nullptr, nullptr, nullptr, q, k, v, bias, sinks, nullptr, 1.0f/sqrtf(hd), il, fa_mask);
cur = ggml_mul(ctx0, cur, ggml_sigmoid(ctx0, build_lora_mm(l.wqkv_gate, inputs[0])));
cur = convolve(build_lora_mm(l.wo, cur), l.attn_post_conv, nullptr, state, history, recr, 2*kv);
// the first layer of each block starts a new residual stream, the earlier one joins the mixed blocks
if (il % hparams.attn_res_block_size == 0) {
blocks.push_back(residual);
blocks.push_back(inpSA);
} else {
cur = ggml_add(ctx0, residual, cur);
cur = ggml_add(ctx0, inpSA, cur);
}
residual = cur;
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, l.ffn_res_query), n_embd, n_tokens);
cur = build_norm(cur, l.ffn_norm, nullptr, LLM_NORM_RMS, il);
cur = build_moe_ffn(cur, l.ffn_gate_inp, nullptr, nullptr, l.ffn_down_exps, l.ffn_exp_probs_b,
hparams.n_expert, hparams.n_expert_used(il), LLM_FFN_SITU, true, hparams.expert_weights_scale,
LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID, il, nullptr, l.ffn_gate_up_exps);
cur = convolve(cur, l.ffn_post_conv, nullptr, state, history, recr, 2*kv + n_embd);
cur = ggml_add(ctx0, residual, cur);
ggml_tensor * ffn_inp = cur;
cb(ffn_inp, "ffn_inp", il);
// MoE
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, layer.ffn_res_query), n_embd, n_tokens);
cur = build_norm(cur, layer.ffn_norm, nullptr, LLM_NORM_RMS, il);
cur = build_moe_ffn(cur,
layer.ffn_gate_inp,
nullptr,
nullptr,
layer.ffn_down_exps,
layer.ffn_exp_probs_b,
hparams.n_expert,
hparams.n_expert_used(il),
LLM_FFN_SITU, true,
hparams.expert_weights_scale,
LLAMA_EXPERT_GATING_FUNC_TYPE_SIGMOID,
il,
nullptr,
layer.ffn_gate_up_exps);
cur = convolve(cur, layer.ffn_post_conv, nullptr, state, history, recr, 2*n_embd_k_gqa + n_embd);
cb(cur, "ffn_out", il);
cur = ggml_add(ctx0, ffn_inp, cur);
cb(cur, "l_out", il);
}
cur = ggml_reshape_2d(ctx0, mix(blocks, cur, model.output_res_query), n_embd, n_tokens);
cur = build_norm(cur, model.output_norm, nullptr, LLM_NORM_RMS, -1);
if (out_ids) {
cur = ggml_get_rows(ctx0, cur, out_ids);
if (inp_out_ids) {
cur = ggml_get_rows(ctx0, cur, inp_out_ids);
}
cb(cur, "result_norm", -1);
res->t_embd = cur;
// lm_head
cur = build_lora_mm(model.output, cur);
ggml_prec_set_acc(cur, GGML_PREC_F32);
const float cap = hparams.f_final_logit_softcapping;
cur = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/cap)), cap);
if (hparams.f_final_logit_softcapping != 0.0f) {
const float cap = hparams.f_final_logit_softcapping;
cur = ggml_scale(ctx0, ggml_tanh(ctx0, ggml_scale(ctx0, cur, 1.0f/cap)), cap);
}
// drop the padded vocabulary
cur = ggml_cont(ctx0, ggml_view_2d(ctx0, cur, hparams.n_vocab_unpadded, cur->ne[1], cur->nb[1], 0));
res->t_logits = cur;
cb(cur, "result_output", -1);
res->t_logits = cur;
}
+8 -1
View File
@@ -2480,7 +2480,14 @@ struct llama_model_inkling2 : public llama_model_base {
graph(const llm_graph_params & params);
ggml_tensor * mix(const std::vector<ggml_tensor *> & blocks, ggml_tensor * cur, ggml_tensor * query);
ggml_tensor * clip(ggml_tensor * cur, ggml_tensor * scale, float threshold);
ggml_tensor * convolve(ggml_tensor * cur, ggml_tensor * weight, ggml_tensor * dynamic, ggml_tensor * state, ggml_tensor * history, llm_graph_input_rs * inp, int64_t offset);
ggml_tensor * convolve(
ggml_tensor * cur,
ggml_tensor * weight,
ggml_tensor * dynamic,
ggml_tensor * state,
ggml_tensor * history,
llm_graph_input_rs * inp,
int64_t offset);
};
std::unique_ptr<llm_graph_context> build_arch_graph(const llm_graph_params & params) const override;
+2
View File
@@ -367,6 +367,8 @@ endif()
llama_build(test-backend-ops.cpp LINK ggml Threads::Threads)
llama_build_and_test(test-backend-tensor.cpp LINK ggml)
llama_build_and_test(test-sleep.cpp LINK ggml)
llama_build_and_test(test-backend-sched.cpp LINK ggml)
if (BUILD_SHARED_LIBS)
llama_build_and_test(test-llama-mmap.cpp ${PROJECT_SOURCE_DIR}/src/llama-impl.cpp ${PROJECT_SOURCE_DIR}/src/llama-mmap.cpp LABEL behavior)
+88 -9
View File
@@ -2830,6 +2830,12 @@ struct test_set_rows : public test_case {
if (type_src == GGML_TYPE_F16) {
err_estimate *= 16.0f;
}
if (type_src == GGML_TYPE_BF16) {
// BF16 values have 8 significant bits, so many of them land exactly on a rounding tie,
// and backends break ties differently (e.g. the CPU q8_0 rounds half to even, CUDA half away from zero).
// Allow 1 in 32 values to be off by one instead of one value in total.
return err_estimate / (0.25f*32.0f);
}
err_estimate /= 0.25f*float(ne[0] * r * ne[2]*nr23[0] * ne[3]*nr23[1]);
return err_estimate;
}
@@ -2860,10 +2866,15 @@ struct test_rope_set_rows : public test_case {
const int n_dims;
const bool yarn;
const bool truncate;
const ggml_type type_a; // type of the rope input
const int n_ctx{512};
std::string vars() override {
return VARS_TO_STR7(type, type_idx, ne_a, mode, n_dims, yarn, truncate);
std::string s = VARS_TO_STR7(type, type_idx, ne_a, mode, n_dims, yarn, truncate);
if (type_a != GGML_TYPE_F32) {
s += "," + VAR_TO_STR(type_a);
}
return s;
}
std::string op_desc(ggml_tensor * t) override {
@@ -2871,17 +2882,27 @@ struct test_rope_set_rows : public test_case {
return "ROPE_SET_ROWS";
}
double max_nmse_err() override {
if (type_a != GGML_TYPE_BF16) {
return 1e-7;
}
// into F16 the reference rounds the BF16 rope result to BF16 first, the fused rope + set_rows does not
// otherwise the limit of a BF16 test_rope: the angle error n_ctx*FLT_EPSILON can flip a BF16 rounding
return type == GGML_TYPE_F16 ? 1e-5 : n_ctx * FLT_EPSILON / 128.0;
}
bool run_whole_graph() override { return true; }
test_rope_set_rows(ggml_type type,
ggml_type type_idx,
std::array<int64_t, 4> ne_a,
int mode,
int n_dims = 128, bool yarn = false, bool truncate = true)
: type(type), type_idx(type_idx), ne_a(ne_a), mode(mode), n_dims(n_dims), yarn(yarn), truncate(truncate) {}
int n_dims = 128, bool yarn = false, bool truncate = true,
ggml_type type_a = GGML_TYPE_F32)
: type(type), type_idx(type_idx), ne_a(ne_a), mode(mode), n_dims(n_dims), yarn(yarn), truncate(truncate), type_a(type_a) {}
ggml_tensor * build_graph(ggml_context * ctx) override {
ggml_tensor * a = ggml_new_tensor_4d(ctx, GGML_TYPE_F32, ne_a[0], ne_a[1], ne_a[2], 1);
ggml_tensor * a = ggml_new_tensor_4d(ctx, type_a, ne_a[0], ne_a[1], ne_a[2], 1);
ggml_set_name(a, "a");
const bool is_mrope = mode & GGML_ROPE_TYPE_MROPE;
@@ -6793,6 +6814,10 @@ struct test_rope : public test_case {
// the f32 rotation angle of a position below n_ctx carries a rounding error of up to n_ctx*FLT_EPSILON radians
// backends compute the angle with different rounding sequences, so their agreement is bounded by that error
const double eps_theta = n_ctx * FLT_EPSILON;
if (type == GGML_TYPE_BF16) {
// with BF16 output that error can also change the rounding: a step of 2^-7 with a chance of about eps_theta/2^-7
return std::max(test_case::max_nmse_err(), eps_theta / 128.0);
}
return std::max(test_case::max_nmse_err(), eps_theta * eps_theta);
}
@@ -9657,6 +9682,33 @@ struct test_fill : public test_case {
}
};
// GGML_OP_SLEEP (for scheduler testing only)
struct test_sleep : public test_case {
const ggml_type type;
const std::array<int64_t, 4> ne;
const int32_t us;
std::string vars() override { return VARS_TO_STR3(type, ne, us); }
// the op only delays, the output aliases the input and must match it exactly
double max_nmse_err() override { return 0.0; }
test_sleep(int32_t us = 100, ggml_type type = GGML_TYPE_F32,
std::array<int64_t, 4> ne = { 10, 10, 4, 3 })
: type(type), ne(ne), us(us) {}
ggml_tensor * build_graph(ggml_context * ctx) override {
ggml_tensor * a = ggml_new_tensor_4d(ctx, type, ne[0], ne[1], ne[2], ne[3]);
ggml_set_name(a, "a");
ggml_tensor * out = ggml_sleep(ctx, a, us);
ggml_set_name(out, "out");
return out;
}
};
// GGML_OP_SOLVE_TRI
struct test_solve_tri : public test_case {
const ggml_type type;
@@ -10868,6 +10920,12 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_set_rows(GGML_TYPE_F16, GGML_TYPE_F16, GGML_TYPE_I32, { 1, 8, 1, 3 }, { 1, 1 }, 2, false));
test_cases.emplace_back(new test_set_rows(GGML_TYPE_F16, GGML_TYPE_F16, GGML_TYPE_I64, { 1, 8, 1, 3 }, { 1, 1 }, 2, true));
test_cases.emplace_back(new test_set_rows(GGML_TYPE_F16, GGML_TYPE_F16, GGML_TYPE_I32, { 1, 8, 1, 3 }, { 1, 1 }, 2, true));
// BF16 source: one case per type covers batch dims, broadcast indices and gaps between rows
for (ggml_type type : all_types) {
test_cases.emplace_back(new test_set_rows(GGML_TYPE_BF16, type, GGML_TYPE_I64, { 256, 11, 1, 1 }, { 2, 3, }, 7, true));
}
test_cases.emplace_back(new test_set_rows(GGML_TYPE_BF16, GGML_TYPE_Q8_0, GGML_TYPE_I32, { 256, 5, 1, 3 }, { 1, 1, }, 1, false));
test_cases.emplace_back(new test_set_rows(GGML_TYPE_BF16, GGML_TYPE_F16, GGML_TYPE_I64, { 1, 8, 1, 3 }, { 1, 1 }, 2, false));
for (int mode : { GGML_ROPE_TYPE_NORMAL, GGML_ROPE_TYPE_NEOX, GGML_ROPE_TYPE_MROPE, GGML_ROPE_TYPE_VISION, GGML_ROPE_TYPE_IMROPE }) {
for (ggml_type type : {GGML_TYPE_F16, GGML_TYPE_F32}) {
@@ -10878,11 +10936,19 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
}
}
test_cases.emplace_back(new test_rope_set_rows(GGML_TYPE_F32, GGML_TYPE_I32, { 128, 32, 8, 1 }, GGML_ROPE_TYPE_IMROPE));
for (int mode : { GGML_ROPE_TYPE_NORMAL, GGML_ROPE_TYPE_NEOX, GGML_ROPE_TYPE_MROPE, GGML_ROPE_TYPE_VISION, GGML_ROPE_TYPE_IMROPE }) {
for (ggml_type type : {GGML_TYPE_F16, GGML_TYPE_BF16}) {
for (int ne2 : {1, 8, 512}) {
test_cases.emplace_back(new test_rope_set_rows(type, GGML_TYPE_I64, { 128, 32, ne2, 1 }, mode, 128, false, true, GGML_TYPE_BF16));
}
}
}
// partial rotation
for (int mode : { GGML_ROPE_TYPE_NORMAL, GGML_ROPE_TYPE_NEOX, GGML_ROPE_TYPE_MROPE, GGML_ROPE_TYPE_IMROPE }) {
for (ggml_type type : {GGML_TYPE_F16, GGML_TYPE_F32}) {
test_cases.emplace_back(new test_rope_set_rows(type, GGML_TYPE_I64, { 256, 4, 8, 1 }, mode, 64));
}
test_cases.emplace_back(new test_rope_set_rows(GGML_TYPE_F16, GGML_TYPE_I64, { 256, 4, 8, 1 }, mode, 64, false, true, GGML_TYPE_BF16));
}
for (int mode : {GGML_ROPE_TYPE_NORMAL, GGML_ROPE_TYPE_NEOX}) {
@@ -12341,6 +12407,9 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_BF16, GGML_TYPE_F32, 16, 16, b, 50, 200, 64));
}
// One very large one to trigger a code path in CUDA requiring >1 pass over expert IDs due to shared memory limits:
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_Q8_0, GGML_TYPE_F32, 16, 16, false, 32, 65536, 256));
// For issue 27873
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_IQ2_XXS, GGML_TYPE_F32, 1, 1, false, 1, 8192, 4096));
@@ -12375,8 +12444,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_TQ1_0, GGML_TYPE_F32, 28, 10, false, 1024, 1, 4096));
test_cases.emplace_back(new test_mul_mat_id(GGML_TYPE_TQ1_0, GGML_TYPE_F32, 128, 8, false, 1024, 1, 2048));
// TODO: disabled until CUDA backend support is added [TAG_MUL_MAT_ID_DUPLICATE_ID]
#if 0
// repeated expert ids within a row
for (ggml_type ta : {GGML_TYPE_F16, GGML_TYPE_Q8_0, GGML_TYPE_Q4_0}) {
for (int n : {9, 16, 33, 64}) {
@@ -12388,7 +12455,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_mul_mat_id_dup(ta, GGML_TYPE_F32, 1025, 10, false, 64, 33, 256, 0, true));
test_cases.emplace_back(new test_mul_mat_id_dup(ta, GGML_TYPE_F32, 2050, 10, true, 64, 33, 256));
}
#endif
for (ggml_type type_a : all_types) {
test_cases.emplace_back(new test_mul_mat_id(type_a, GGML_TYPE_F32, 4, 2, false, 64, 16, 3*ggml_blck_size(type_a)));
@@ -12689,8 +12755,17 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
}
}
// BF16: each mode forward on a strided view and backward on an offset view, with and without freq factors, and one in place
for (int mode : {GGML_ROPE_TYPE_NORMAL, GGML_ROPE_TYPE_NEOX, GGML_ROPE_TYPE_MROPE, GGML_ROPE_TYPE_IMROPE, GGML_ROPE_TYPE_VISION}) {
for (bool ff : {false, true}) {
test_cases.emplace_back(new test_rope(GGML_TYPE_BF16, {128, 32, 2, 1}, 128, mode, 512, 1.4245f, 0.7465f, 1.4245f, ff, 1, true));
test_cases.emplace_back(new test_rope(GGML_TYPE_BF16, {128, 32, 2, 1}, 128, mode, 512, 1.4245f, 0.7465f, 1.4245f, ff, 2, false));
}
test_cases.emplace_back(new test_rope(GGML_TYPE_BF16, {128, 32, 2, 1}, 128, mode, 512, 1.4245f, 0.7465f, 1.4245f, false, 0, true, true));
}
// rotated dims window at an offset (ggml_rope_set_offset), not supported for vision mode
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16}) {
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16}) {
for (bool fw : {true, false}) { // fw == forward
for (bool ff : {false, true}) {
test_cases.emplace_back(new test_rope(type, {128, 32, 2, 1}, 32, GGML_ROPE_TYPE_NORMAL, 512, 1.4245f, 0.7465f, 1.4245f, ff, 0, fw, false, 32));
@@ -12996,6 +13071,10 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_fill(-152.0f, GGML_TYPE_F16, { 800, 600, 4, 4 }));
test_cases.emplace_back(new test_fill(3.5f, GGML_TYPE_F16, { 2048, 512, 2, 2 }));
test_cases.emplace_back(new test_sleep(0));
test_cases.emplace_back(new test_sleep(10, GGML_TYPE_F16, { 128, 4, 2, 2 }));
test_cases.emplace_back(new test_sleep(10, GGML_TYPE_F32, { 128, 4, 2, 2 }));
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16}) {
test_cases.emplace_back(new test_diag(type));
test_cases.emplace_back(new test_diag(type, { 79, 1, 19, 13 }));
@@ -14077,7 +14156,7 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
}
for (bool fw : {true, false}) { // fw == forward
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16}) {
for (ggml_type type : {GGML_TYPE_F32, GGML_TYPE_F16, GGML_TYPE_BF16}) {
for (bool ff : {false, true}) { // freq_factors
for (float v : { 0, 1 }) {
test_cases.emplace_back(new test_rope(type, {128, 32, 512, 1}, 128, GGML_ROPE_TYPE_NORMAL, 512, 1.0f, 0.0f, 1.0f, ff, v, fw)); // llama 7B
File diff suppressed because it is too large Load Diff
+34
View File
@@ -170,6 +170,40 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
}
}
{
std::vector<ggml_tensor *> batch;
ggml_backend_buffer_ptr scatter_buf = alloc_tensors(ctx.get(), buft, { 64 }, batch);
if (!scatter_buf) {
return false;
}
batch.resize(128, batch[0]);
uint8_t input[32], expected[256], out[256];
for (size_t i = 0; i < sizeof(input); i++) {
input[i] = (uint8_t) (i + 1);
}
std::vector<size_t> offsets(batch.size()), sizes(batch.size(), 1), data_offsets(batch.size());
for (bool overlap : { false, true }) {
memset(expected, 0, sizeof(expected));
ggml_backend_tensor_set(batch[0], expected, 0, sizeof(expected));
for (size_t i = 0; i < batch.size(); i++) {
offsets[i] = 2*((i*5) % batch.size()) + 1;
data_offsets[i] = i % sizeof(input);
}
// Repeated sources are valid; overlapping destinations must retain copy order.
if (overlap) {
offsets.back() = offsets.front();
}
for (size_t i = 0; i < batch.size(); i++) {
expected[offsets[i]] = input[data_offsets[i]];
}
ggml_backend_tensor_set_n(batch.data(), input, offsets.data(), sizes.data(), data_offsets.data(), batch.size());
ggml_backend_tensor_get(batch[0], out, 0, sizeof(out));
if (memcmp(out, expected, sizeof(out)) != 0) {
return false;
}
}
}
return true;
}
+154
View File
@@ -0,0 +1,154 @@
// Tests for GGML_OP_SLEEP, on every backend that supports it. Three properties are checked:
// - the op only delays: input = output
// - the op does not return early: a chain of n sleeps must take at least n*us
// Only a lower bound is asserted for the duration. A busy-wait cannot finish early, so that bound is
// deterministic, whereas an upper bound would be flaky on a loaded machine.
// Known limitation: Vulkan does not specify the tick rate of the shader clock, see ggml_vk_sleep. A driver
// whose clock runs slower than assumed could undershoot the request and fail this bound.
#include "ggml.h"
#include "ggml-alloc.h"
#include "ggml-backend.h"
#include "ggml-cpp.h"
#include <cinttypes>
#include <cstdio>
#include <cstring>
#include <random>
#include <vector>
enum test_result {
TEST_OK,
TEST_FAIL,
TEST_SKIP,
};
static test_result test_sleep(ggml_backend_t backend, ggml_type type, int64_t ne, int32_t us, int n_nodes) {
const size_t graph_size = n_nodes + 16;
ggml_init_params params_static = {
/*.mem_size =*/ ggml_tensor_overhead(),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ggml_context_ptr ctx_static(ggml_init(params_static));
ggml_tensor * x = ggml_new_tensor_1d(ctx_static.get(), type, ne);
ggml_set_name(x, "x");
ggml_set_input(x);
ggml_init_params params_compute = {
/*.mem_size =*/ n_nodes*ggml_tensor_overhead() + ggml_graph_overhead_custom(graph_size, false),
/*.mem_buffer =*/ nullptr,
/*.no_alloc =*/ true,
};
ggml_context_ptr ctx_compute(ggml_init(params_compute));
ggml_cgraph * gf = ggml_new_graph_custom(ctx_compute.get(), graph_size, false);
ggml_tensor * out = x;
for (int i = 0; i < n_nodes; i++) {
out = ggml_sleep(ctx_compute.get(), out, us);
}
ggml_set_name(out, "out");
ggml_build_forward_expand(gf, out);
// the backends log to stdout while computing, so the result line is only printed once it is complete
char desc[128];
snprintf(desc, sizeof(desc), " %-6s %-4s ne=%-9" PRId64 " us=%-7d n_nodes=%d ",
ggml_backend_name(backend), ggml_type_name(type), ne, us, n_nodes);
test_result result = TEST_OK;
if (!ggml_backend_supports_op(backend, out)) {
printf("%snot supported\n", desc);
result = TEST_SKIP;
} else {
ggml_backend_buffer_ptr buf_static(ggml_backend_alloc_ctx_tensors(ctx_static.get(), backend));
// random bytes rather than random floats, so that bit patterns a value-wise copy could
// mangle, such as NaNs and denormals, are covered as well
std::vector<uint8_t> data_in(ggml_nbytes(x));
std::mt19937 rng(1234);
for (size_t i = 0; i < data_in.size(); i++) {
data_in[i] = rng() & 0xFF;
}
ggml_backend_tensor_set(x, data_in.data(), 0, data_in.size());
ggml_gallocr_ptr galloc(ggml_gallocr_new(ggml_backend_get_default_buffer_type(backend)));
ggml_gallocr_alloc_graph(galloc.get(), gf);
// warm up, so that one-time costs such as lazy CUDA module loading are not timed
ggml_backend_graph_compute(backend, gf);
const int64_t t_start_us = ggml_time_us();
const ggml_status status = ggml_backend_graph_compute(backend, gf);
const int64_t t_us = ggml_time_us() - t_start_us;
std::vector<uint8_t> data_out(ggml_nbytes(out));
ggml_backend_tensor_get(out, data_out.data(), 0, data_out.size());
const int64_t t_min_us = (int64_t) n_nodes * us;
const bool ok_status = status == GGML_STATUS_SUCCESS;
const bool ok_alias = out->data == x->data;
const bool ok_data = memcmp(data_in.data(), data_out.data(), data_in.size()) == 0;
const bool ok_time = t_us >= t_min_us;
printf("%selapsed=%7" PRId64 " us (>= %6" PRId64 ") alias: %-4s data: %-4s time: %s\n",
desc, t_us, t_min_us, ok_alias ? "OK" : "FAIL", ok_data ? "OK" : "FAIL", ok_time ? "OK" : "FAIL");
if (!ok_status) {
printf(" compute failed: %s\n", ggml_status_to_string(status));
}
result = ok_status && ok_alias && ok_data && ok_time ? TEST_OK : TEST_FAIL;
}
return result;
}
int main() {
ggml_time_init();
ggml_backend_load_all();
int n_ok = 0;
int n_fail = 0;
int n_skip = 0;
for (size_t i = 0; i < ggml_backend_dev_count(); i++) {
ggml_backend_dev_t dev = ggml_backend_dev_get(i);
ggml_backend_ptr backend(ggml_backend_dev_init(dev, nullptr));
if (backend == nullptr) {
printf(" failed to initialize %s\n", ggml_backend_dev_name(dev));
n_fail++;
continue;
}
// a zero duration must still be a valid no-op, longer ones are checked against the clock,
// and the last case verifies that a chain of sleeps accumulates instead of collapsing into one
const struct {
ggml_type type;
int64_t ne;
int32_t us;
int n_nodes;
} cases[] = {
{ GGML_TYPE_F32, 1, 0, 1 },
{ GGML_TYPE_F32, 4096, 10000, 1 },
{ GGML_TYPE_F16, 1024, 10000, 1 },
{ GGML_TYPE_F32, 1 << 20, 5000, 4 },
};
for (const auto & c : cases) {
switch (test_sleep(backend.get(), c.type, c.ne, c.us, c.n_nodes)) {
case TEST_OK: n_ok++; break;
case TEST_FAIL: n_fail++; break;
case TEST_SKIP: n_skip++; break;
}
}
}
printf("%d passed, %d failed, %d skipped\n", n_ok, n_fail, n_skip);
return n_fail > 0;
}
+1
View File
@@ -644,6 +644,7 @@ static void mtmd_owned_llama_set_tensor_data(ggml_tensor * t, void * user_data)
return name.size() >= n && name.compare(name.size() - n, n, suffix) == 0;
};
const bool optional = ends_with("scale") || ends_with(".bias") || name == "dense_2.weight" ||
name == "cls.output.weight" || // Qwen3 reranking head, unused by YuE2 NAR
name == "rope_freqs.weight" || // granite, voxcpm2 residual LM
name == "token_embd.weight" || name == "output.weight"; // inputs are embedding rows, outputs are hidden states
if (!optional || t->type != GGML_TYPE_F32) {
+4
View File
@@ -353,6 +353,8 @@ struct server_slot {
return false;
}
const int64_t t_start = ggml_time_us();
const size_t cur_size_tgt = llama_state_seq_get_size_ext(ctx_tgt, id, LLAMA_STATE_SEQ_FLAGS_NONE);
const size_t cur_size_dft = ctx_dft ? llama_state_seq_get_size_ext(ctx_dft, id, LLAMA_STATE_SEQ_FLAGS_NONE) : 0;
@@ -374,6 +376,8 @@ struct server_slot {
return false;
}
SLT_TRC(*this, "prompt cache save took %.2f ms\n", (ggml_time_us() - t_start) / 1000.0);
return true;
}
+4
View File
@@ -208,6 +208,8 @@ bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tok
if (it_best != states.end()) {
SRV_TRC(" - found better prompt with f_keep = %.3f, f_sim = %.3f\n", f_keep_best, f_sim_best);
const int64_t t_start = ggml_time_us();
{
auto & data = it_best->data.main;
@@ -252,6 +254,8 @@ bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tok
prompt = std::move(it_best->prompt);
states.erase(it_best);
SRV_TRC("slot %d: prompt cache restore took %.2f ms\n", id_slot, (ggml_time_us() - t_start) / 1000.0);
}
return true;