Erik Scholz 56e659a0b2 fix perplexity after c-api refactor (#390) ...

* preallocate a buffer of fitting size for tokenization (utils.cpp)

* don't create a new std::string (especially here, where it's usually large)

2023-03-22 18:09:38 +02:00

7.3 KiB

Raw Permalink Blame History

View Raw