* hexagon: use non-host bufs by default and make the backend fully async
* hex-hb: remove optional hostbuf support and fix async copy
* hex-unary: relax supported unary check
* hex-bufs: use same get_alignment for host bufs
* snapdragon: bump android_platform to 34
* hex-rows: super hacky get/set rows for q8_0
* hex-get-rows: fix q8_0
* hex-get-rows: supprot for f16 and cleanup for q8_0
* hex-get-rows: generic macros and specialized thread funcs
* hex-get-rows: add DMA pipeline, vtcm_layout and kernel params
* hex-set-rows: fix q8_0 support, add dma and tracing
* hex-tests: override nmse threshold for HTP of Q8_0 quants
* hex-fa: add support for Q8_0 with inplace dequantizers
* hex-get-rows: simplify type dispatch
* hex-rows: simplify GET/SET_ROWS DMA pipeline
* hex-async: add events, set/get-tensor-async and rest of the async api support
* hex-repack: use slice instead of expert in repack functions
* hex-cpy: update event/async-cpy logging
* hex-set-rows: optimize smaller tensors
* hex-geglu: fix perf regression with larger tensors
* hex-get-rows: add missing header
* hex-set-rows: add missing header
* hex-bufs: ressurect GGML_HEXAGON_HOSTBUF but disable it by default
* hexagon: do not reject ops with non-heaxon buffers
* hex-get-rows: apply >=32 restriction only for q8_0
* hex-res: bump vtcm acquire timeout to 10 seconds
* hex-bufs: add support for cloning buffers between sessions to speed up tensor copies
* hex-async: rework event recording and batch flushing and integrate with meta backend
* hex-bufs: improved handling of repacked tensors
* hex-repack: handle get_tensor_2d offsets
* hex-dev: add support for devices with multiple NPUs
* hex-sync: add support for sync tokens to synchronize npu devices for async splits
* hex-mmap: cleanup mmap calls and add a retry for robustness
* hex-sync: add failsafe if sync wait gets stuck
* hex-sync: use sync_seq to check for completed events
* hex-sync: rotate tokens for extra robustness
* hex-devs: add supprot for legacy device names for now
* hex-bufs: add support for auto-cloning buffers from diff sessions
* hex-fusion: simplify and optimize htp-opnode fusion handling
* hex-sync: override opnode name so that it shows up in the profiles
* hex-trace: update scripts to handle multiple devices
* hex-sync: bump the size of the opbatch queue and number of sync tokens
* hex-cpy-sync: do not explicitly flush opbatches in cpy_tensor_async and add support for cpy-dma
* hex-sync: add graph-flush threshold to avoid single op batches
* hex-sync: add sync_peer so that we can flush peers we depend on during cross-device ops
* hex-bufs: introduce tensor->extra and shadow_bufs for repacking
* hex-l2: flush tiny tensors inline
* hex-sync: use explicit l2flush for sync tokens
* hex-extra: track weight flags via tensor extra
* hex-fence: rename sync to fence
* hex-repack: proper handling of set-tensor-2d in the shadow_buf
* hex-trace: remove obsolete opstage mask that we used for profiling
* hex-env: remove obsolete use_hmx variable
* hexagon: new unified run.py and build.py and updated docs
* snapdragon: update run script to auto-escapt test-backend-op -p argument
* hex-scripts: fix trailing spaces
* hex-scripts: fix flake8 warnings
* snapdragon: cleanup dst lib/bin dirs before copying new build
* hex-ops: add support for allreduce
* hex-ar: improved allreduce with dma pipeline
* hex-ar: align macros
* hex-ar: consistent use of fence_seq
* hex-ar: add AR_SELECT env var to select ALLREDUCE kernel or fallback
* hex-ar: add proper synchronize handling for ALLREDUCE
* hex-opbatch: looks like we now just rely on backend.synchronise to flush the batches, no need to flush them by threshold
* hex-ar: bump block size to improve dma efficiency
* hex-ar: fused ALLREDUCE+ADD
* hex-ar: cleaner fence buffer management
* hex-ar: futher allreduce tweaking to remove race conditions
* hex-ar: add simple solver and remove non-dma kernels
* hex-ar: add row-broadcast to fuse with bias ADD
* hex-fence: pass seq numbers via op_params
* hex-ar: allow for both entry/exit seq for completing entry wait
* hex-ar: align macros
* hex-ar: do not refetch broadcast row
* hex-fusion: move all fusion into opbatch::add_op for consistency with ALLREDUCE and things
* hex-fusion: fix incorrect MUL_MAT reordering
* hex-mm: make fused 2x and 3x matmuls more generic
* hex-fusion: move tensor fusion tagging to graph_compute
* hexagon: make sure to copy tensor->extra by value
* hex-get-rows: fix offset calc with row-chunking
* hex-repack: get_tensor_2d fixes for non-zero offsets
* snapdragon: make profile/trace scripts more robust and donot mix stdout/stderr by default
* hex-devices: use legacy device nameing by default to ease the transition
* hex-devices: hardcode CDSP domain IDs for current devices for now
* hex-optrace: improve multi-NPU timestamp alignment and overall handling of cycle values
* hex-optrace: more robust handling of the fence events
* metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill
* metal: drop scalar SSD path; MMA + sequential tail
* drop WIP ssm scan test noise
* remove state_from_dst and rename CS and NSG constants
* remove unrelated added whitespace padding
* added clarity to mma_tokens calculation
* added clarity to use_mma bool checks
* added comments to metal ssd op constants for clarity
* reserve K tokens for sequential kernel rollback snapshots
* reset concurrency between mma and seq tail
* remove print args no longer used
* fixed comment to no longer point to specific line
* add FC_SSM_SCAN so seq path skips token offlset unless it's mma tail
* added changes to new ssm.metal for rebase after ggml-metal.metal refactor
* specialize ssm_scan tail with a template instead of a function constant
---------
Co-authored-by: dpantaleoni <dominikpantaleoni@gmail.com>
Co-authored-by: forforever73 <690105611@qq.com>
* rpc: support apple RDMA as an RPC transport
* remove set_tensor micro optimization, rpc socket pinning per CR
* remove transparent reconnect
* trigger apple builds on RPC changes
---------
Co-authored-by: Ryan Churaman <rschu@meta.com>
* metal : null-check ggml_metal_buffer_init result to avoid OOM crash
ggml_backend_metal_buffer_type_alloc_buffer used the result of
ggml_metal_buffer_init without checking for NULL. ggml_metal_buffer_init
returns NULL when the underlying Metal allocation fails (e.g. an
out-of-memory condition), and the following ggml_metal_buffer_is_shared(res)
call dereferences it, turning a recoverable allocation failure into a hard
crash (EXC_BAD_ACCESS). This is easy to hit on memory-constrained devices
such as iOS when a model/context exceeds the available Metal budget.
Log the failure using the existing GGML_LOG_ERROR convention and return
NULL so the allocator surfaces a diagnosable error up the stack instead of
crashing.
* cont : fix log
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Bump version to 1.3.9
* Fix JFKReader's format
* Add Whisper::Context#free
* Add test for Whisper::Context#free
* Add Whisper::Context#free to RBS
* Add Whisper::VAD::Context#free
* Add test for Whisper::VAD::Context#free
* Add Whisper::VAD::Context#free to RBS
* Add Parakeet::Context#free
* Add test for Parakeet::Context#free
* Add Parakeet::Context#free to RBS
* Check MemoryView format more strictly
* Fallback MemoryView flags to SIMPLE when MemoryView not available
* Add changelog entries
* Update document on callbacks
* Remove whitespace
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
---------
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
whisper_full()/whisper_full_with_state() only compute the mel spectrogram when
n_samples > 0. For n_samples == 0 on a freshly allocated state the mel is never
touched, but struct whisper_mel had no member initializers, so n_len / n_len_org
/ n_mel were indeterminate heap garbage (the state is allocated with
new whisper_state). seek_end is derived from that garbage and, depending on it,
the call either quietly returns 0 or runs the encoder with garbage dimensions
over an empty (NULL) mel buffer, dereferencing address 0 in the mel copy loop.
Give whisper_mel default member initializers so a never-computed mel reads as
0 frames and the n_samples == 0 case deterministically takes the existing
too-short path.
Fixes#3978
* metal : per-device tuned (Q, NE) for flash-attn vec (llama/25750)
* rebase Q-generic FA vec body from 01dc93607 (llama/23114)
* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)
* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback
* add FA vec (Q,NE) perf sweep
* fill tuning result
* fold family table into a per-family representative SKU
* refactor tuning result format
* extend FA vec tuning to quantized KV caches
* sync fa vec tuner bucketing with runtime, use pointwise tuning regret
* update tuned table
* format and cleanup
* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active
* add device id -> token lookup for the offline tuning tool
* add ggml-metal-tuning skeleton
* add op-agnostic perf cell + median timing for the tuner
* add FA-vec graph build + tensor init to the tuner
* tools : add FA-vec (Q,NE) sweep, compression and table emit
* cool down and re-measure the dirty window on thermal drift
* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice
* tools : document the Metal tuner, point the table comment at it
* abort on unknown KV type, single-source fa_vec_legal_ne
* cleanup
* honor -o in the FA vec (Q,NE) slice
* retune FA-vec (Q, NE) under a pointwise no-harm gate
* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* DSV4: sm tensor
* set coarser granularity for head splits
* fix dspark
* add model saving for dsv4 + allow dflash to return on specific device
* add comment about dsv4 seq_rm
* simplify
* add shared expert delayed allreduce
* remove special test for dsv4
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
* whisper : guard null source in buffer loader read callback
whisper_init_from_buffer_with_params_no_state installs a read callback that
copies from buf->buffer + current_offset. When the buffer is exhausted (or the
supplied buffer is empty), size_to_copy is 0 and the source pointer can be null;
passing a null pointer to memcpy is undefined behavior even for a zero-length
copy (UBSan: 'null pointer passed as argument 2' at the memcpy). Loading a
crafted/short model through the buffer loader could hit this.
Skip the memcpy when there is nothing to copy. Loading from a null/empty or
truncated buffer now fails gracefully (returns NULL) with no UB.
This addresses bug 1 of #3879. Bug 2 (integer overflow when sizing the mel
filter buffer) is covered by the open PR #3780.
* fixup! whisper : guard null source in buffer loader read callback
---------
Co-authored-by: Ben Younes <2910651+ousamabenyounes@users.noreply.github.com>
This commit updates the OpenVINO model conversion script and its
dependencies as they were currently not working.
The first issue was an import that needed updating for fix the following
error:
```console
(openvino_conv_env) $ python convert-whisper-to-openvino.py --model base.en
Traceback (most recent call last):
File "/home/danbev/work/ai/whisper-work/models/convert-whisper-to-openvino.py", line 6, in <module>
from openvino.runtime import serialize
ModuleNotFoundError: No module named 'openvino.runtime'
```
And after that there was a missing dependency:
```console
ModuleNotFoundError: No module named 'onnxscript'
```
With the changes in this commit I was able to successfully convert the model
and run the inference using OpenVINO.
* scripts : add release.sh script [no ci]
This commit adds a release.sh script similar to what ggml and llama.cpp
have to prepare a release.
* scripts : use sed_inplace [no ci]
This commit updates the README.md to include a release badge.
The motivation for this is that we have been manually updating the
Stable link in this document for releases. Removing this means that we
don't have to touch this file and only update CMakeLists.txt.
I also removed the link to the Roadmap was it felt out of place and it
has not been kept up to date so I hope that is alright.
I'll follow up with a PR to introduce a release script that will help
assist the creation of releases (updateing the version and creating the
release PR etc).
This commit adds the --parallel flag to the cmake build command to
speed up compilation. This will hopefully help a little with CI runs
even though the are often limited to 2 cores.
Shawn Gu