mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-09-22 22:17:28 -05:00
bri-prism
9a9f939b80
* metal: add F16 input to the FWHT The Metal FWHT kernel accepts F32 input only. This change makes the source type a template parameter, so the kernel reads an F16 source directly instead of requiring a converted copy. The F32 instantiations are unchanged. The pipeline name now carries the source type, and supports_op accepts an F16 src1 for the Hadamard hint at the four sizes the kernels cover. Every other F16 src1 path still goes through ggml_metal_supports_mul_mat_op. These are the test cases mentioned in #27779. test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 16/16, MUL_MAT 1265/1265. * metal: ask the same FWHT question in supports_op and the dispatch supports_op admitted an F16 src1 on the type, the hint and the width alone, but the dispatch also requires src1 and dst to be contiguous and the same shape. A Hadamard hinted MUL_MAT that passed the first and failed the second reached the generic path, which has no F32 src0 by F16 src1 kernel, and aborted on a nil pipeline: kernel not found in any metal library: base = 'kernel_mul_mv_f32_f16_4' ggml_metal_encoder_set_pipeline: nil Metal pipeline ggml_metal_use_fwht now holds the whole condition and both callers use it, so they cannot drift apart again. The added test case has src1 and dst of different shapes, which aborted before this change and is declined by the Metal backend after it. * metal: branchless butterfly select in the FWHT simdgroup kernel Review suggestion. Replaces the ternary in the shuffle stages with val2 - val + 2*((lane & i) == 0)*val, which is the same value without the select. Measured on M5 Pro, interleaved A/B, five rounds, first discarded, on a Hadamard matmul with block 512 and 65536 rows so the kernel rather than the launch dominates: 1324.6 us before, 1285.0 us after, a 3.0% gain, and faster in every round. At the shapes already in the perf suite the op runs 1.6 to 3.9 us against a 1.6 us launch floor, so the difference is not visible there. FOR_UNROLL on the same loops was also measured and made no difference, the delta changing sign between rounds, so it is not included. * metal: move the FWHT dispatch predicates to ggml-metal-common Review feedback. ggml_metal_use_fwht and ggml_metal_fwht_supported_size were static inline in ggml-metal-device.h. They now follow the ggml_metal_op_mul_mat_use_mm pattern: declared in ggml-metal-common.h and implemented in ggml-metal-common.cpp, which is already the home for helpers shared between supports_op and the op dispatch. The predicate is named ggml_metal_op_mul_mat_use_fwht to sit alongside the _use_mm pair it parallels. This also fixes the macos-latest-arm64 build. The header needed ggml-impl.h for ggml_get_op_params_i32, but ggml-metal-device.h is reached from tools/tuning through ggml-metal-tuning.h, and that target does not have ggml/src on its include path. ggml-metal-common.cpp already includes ggml-impl.h, so the accessor is used normally there and the header goes back to needing nothing extra. * metal: keep the FWHT size check internal and group the dispatch helpers Applies the patch from the review. ggml_metal_fwht_supported_size becomes static in ggml-metal-common.cpp since nothing outside it needs the size list, which also drops stdint.h from the header again, and ggml_metal_op_mul_mat_use_fwht joins the existing _use_mm declarations under their shared comment instead of carrying its own block. * tests: drop the mismatched-shape Hadamard case I added a case with m != k to cover an abort, but the hint is a promise that src0 is a Hadamard matrix, so src0 is square and dst has the same shape as src1. Every other case in the suite holds to that. The case was not a valid op, and on CPU it compared the FWHT against a real matmul of a non-square src0, which cannot agree. The supports_op and dispatch conditions still come from one predicate, which is what keeps them from disagreeing on contiguity.
llama.cpp
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
Quick start
A few options to get llama.cpp installed on your machine:
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
|
|
|
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
Supported backends
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
Documentation
Tools
Development
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
Contributing
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
Acknowledgements
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain