mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-09-29 17:37:39 -05:00

Files

T

History

Xuan-Son Nguyen f1ea206218 batch: migrate speculative, mtmd and server to batch_ext (#29385 )

* adapt common

* add common_batch

* wip

* wip: spec

* cont

* common_speculative_process

* server_batch to use common_batch

* rm some stale calls

Assisted-by: Claude Fable 5.1

* migrate mtmd

* handle imrope, handle return val of add()/add_embd()

* add spec zeros vector

* add warning on zero fill path

2026-09-28 19:52:45 +02:00

CMakeLists.txt

libs : rename libcommon -> libllama-common (#21936 )

2026-04-17 11:11:46 +03:00

README.md

spec : update speculative-simple (#26904 )

2026-08-11 19:52:12 +03:00

speculative-simple.cpp

batch: migrate speculative, mtmd and server to batch_ext (#29385 )

2026-09-28 19:52:45 +02:00

README.md

llama.cpp/examples/speculative-simple

Demonstration of basic greedy speculative decoding

# spec-type draft-simple
./bin/llama-speculative-simple \
        -hf  ggml-org/Qwen3-8B-Base-GGUF:Q8_0 \
        -hfd ggml-org/Qwen3-0.6B-Base-GGUF \
        -p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-simple --spec-draft-n-max 7 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4

# spec-type draft-mtp
./bin/llama-speculative-simple \
        -hf ggml-org/Qwen3.6-27B-GGUF:Q8_0 \
        -p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-mtp --spec-draft-n-max 3 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4

# spec-type draft-mtp (with shared KV cache)
# note: this model needs a <s> token at the start to somewhat work without the chat template
./bin/llama-speculative-simple \
        -hf ggml-org/Gemma-4-31B-it-GGUF:Q8_0 \
        -p "<s>Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-mtp --spec-draft-n-max 3 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4

# spec-type draft-eagle3
./bin/llama-speculative-simple \
        -hf ggml-org/gpt-oss-20b-GGUF \
        -p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-eagle3 --spec-draft-n-max 3 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4

# spec-type draft-dflash
./bin/llama-speculative-simple \
        -hf ggml-org/Qwen3-8B-GGUF \
        -p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-dflash --spec-draft-n-max 7 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4

# spec-type draft-dspark
./bin/llama-speculative-simple \
        -hf ggml-org/Qwen3-8B-GGUF \
        -p "Here is a quick sort implementation in C++. Just code, no comments:\n\n#include" \
        --spec-type draft-dspark --spec-draft-n-max 7 -ngld 99 --color on \
        -n 256 --temp 0 --top-k 1 --seed 42 -ngl 99 -lv 4