upstream-archive/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-08-06 10:00:48 -05:00
Code Issues Packages Projects Releases Wiki Activity

tests : add sample usage

Browse Source
This commit is contained in:
Georgi Gerganov
2024-04-26 13:43:54 +03:00
parent aeafb43ed7
commit e1b2bf783e
2 changed files with 14 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tests/test-tokenizer-0-bpe.py
View File

@@ -1,4 +1,11 @@
# tests with BPE tokenizer
#
# sample usage:
#
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/Meta-Llama-3-8B-Instruct/
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/falcon-7b/
# python3 tests/test-tokenizer-0-bpe.py ~/Data/huggingface/deepseek-coder-6.7b-instruct/
#
import argparse

7
tests/test-tokenizer-0-spm.py
View File

@@ -1,4 +1,11 @@
# tests with SPM tokenizer
#
# sample usage:
#
# python3 tests/test-tokenizer-0-spm.py ~/Data/huggingface/Llama-2-7b-hf/
# python3 tests/test-tokenizer-0-spm.py ~/Data/huggingface/CodeLlama-34b-Instruct-hf/
#
import argparse