upstream-archive/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-10-01 10:27:30 -05:00
Code Issues Packages Projects Releases Wiki Activity
Files
b0aca3c6539e2dd55ea510bbb79591852a4d4b81
llama.cpp/models
T
History
e-mon b8f96c3e82 common : add LLM-jp-4.1 Harmony dialect handler (#29681) 
LLM-jp-4.1 uses the GPT-OSS format, but its tokenizer decodes a space
after every special token and parallel tool calls are separated by
<|end|>. The GPT-OSS handler rejects this output, so add a dedicated
handler, selected by the chat_format=llm-jp-harmony-v1 declaration in
the chat template.

Assisted-by: Claude Fable 5.1
2026-10-01 08:33:55 +03:00
..
templates
common : add LLM-jp-4.1 Harmony dialect handler (#29681)
2026-10-01 08:33:55 +03:00
.editorconfig
gguf : new file format with flexible meta data (beta) (#2398)
2023-08-21 23:07:43 +03:00
ggml-vocab-aquila.gguf
Work on the BPE tokenizer (#3252)
2023-10-03 09:16:26 +02:00
ggml-vocab-baichuan.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00
ggml-vocab-bert-bge.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-bert-bge.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-bert-bge.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf
command-r : add BPE pre-tokenization (#7063)
2024-05-05 08:19:30 +03:00
ggml-vocab-command-r.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-command-r.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-deepseek-coder.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-coder.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-deepseek-llm.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-deepseek-llm.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-falcon.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-falcon.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-gemma-4.gguf
vocab: add gemma4 tokenizer tests, fix edge case (#21534)
2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.inp
vocab: add gemma4 tokenizer tests, fix edge case (#21534)
2026-04-09 11:41:14 +02:00
ggml-vocab-gemma-4.gguf.out
vocab: add gemma4 tokenizer tests, fix edge case (#21534)
2026-04-09 11:41:14 +02:00
ggml-vocab-gpt-2.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-gpt-2.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-2.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-gpt-neox.gguf
Add more tokenizer tests (#3742)
2023-10-24 09:17:17 +02:00
ggml-vocab-llama-bpe.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-llama-bpe.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-llama-bpe.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-llama-spm.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-llama-spm.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-mpt.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-mpt.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-nomic-bert-moe.gguf
tests : improve UGM tokenizer test coverage (#13773)
2025-05-25 16:22:29 +02:00
ggml-vocab-phi-3.gguf
Per token attributes (#7685)
2024-06-04 09:17:17 +02:00
ggml-vocab-phi-3.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-phi-3.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf
llama : add BPE pre-tokenization for Qwen2 (#7114)
2024-05-08 15:06:43 +03:00
ggml-vocab-qwen2.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-qwen2.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-qwen35.gguf
unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)
2026-05-14 11:03:40 +02:00
ggml-vocab-qwen35.gguf.inp
unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)
2026-05-14 11:03:40 +02:00
ggml-vocab-qwen35.gguf.out
unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr… (#22110)
2026-05-14 11:03:40 +02:00
ggml-vocab-refact.gguf
tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)
2024-05-04 08:32:32 +03:00
ggml-vocab-refact.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-refact.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf
llama : fix BPE pre-tokenization (#6920)
2024-04-29 16:58:41 +03:00
ggml-vocab-starcoder.gguf.inp
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00
ggml-vocab-starcoder.gguf.out
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
2025-05-30 12:24:37 +02:00