llama.cpp/utils.cpp at b2de7f18dfbb93463eeb5b4392117bbe82d5bd1b

mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-08-04 00:50:47 -05:00

Files

thement c9f670a177 Implement non-greedy tokenizer that tries to maximize token lengths (#242 )

* Implement non-greedy tokenizer that tries to maximize token lengths

* Insert single space in front of the prompt

- this is to match original llama tokenizer behavior

---------

Co-authored-by: Jakub Horak <jakub.horak@ibawizard.net>

2023-03-17 21:05:58 +01:00

19 KiB

Raw Blame History

View Raw

19 KiB Raw Blame History

19 KiB

Raw Blame History