mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-08-04 09:00:48 -05:00
6b36c23056
* docs : center badges and links, remove Hot topics - Use <div align="center"> for GitHub-compatible centering - Add dev branches and compile times links - Remove Hot topics section Assisted-by: llama.cpp:Qwen3.6-27B * readme : remove sections * docs : center badges, remove Hot topics, extract sections, remove tools - Use <div align="center"> for GitHub-compatible centering - Add dev branches and compile times links - Add lib llama API and llama-server REST API links - Remove Hot topics section - Remove Recent API changes section - Extract XCFramework section into docs/xcframework.md - Extract Completions section into docs/completions.md - Extract Obtaining and quantizing models into docs/models.md - Remove tools usage sections (llama-cli, llama-server, etc.) - Move Contributing section to the end Assisted-by: llama.cpp:Qwen3.6-27B * cont : arrange links * cont : fix ws * cont : remove seminal papers * cont : change sample model * cont : trim-down contributing section * cont : sort backends alphabetically * cont : words * cont : add fig captions * docs : models words * readme : shorter caption * cont : fix typo * cont : add window frame to screenshot
27 lines
1.9 KiB
Markdown
27 lines
1.9 KiB
Markdown
# Obtaining and quantizing models
|
|
|
|
The [Hugging Face](https://huggingface.co) platform hosts [thousands of models](https://huggingface.co/models?library=gguf&sort=trending) compatible with `llama.cpp`:
|
|
|
|
- [Trending](https://huggingface.co/models?library=gguf&sort=trending)
|
|
|
|
You can use any `llama.cpp`-compatible model from [Hugging Face](https://huggingface.co/) using this CLI argument: `-hf <user>/<model>[:quant]`. For example:
|
|
|
|
```sh
|
|
llama cli -hf ggml-org/gemma-3-1b-it-GGUF
|
|
```
|
|
|
|
You can use the same CLI invocation to download from other sites, by pointing the `MODEL_ENDPOINT` environment variable to an endpoint compatible with the Hugging Face API.
|
|
`llama.cpp` can also run models you have downloaded locally to your filesystem.
|
|
|
|
After downloading a model, use the CLI tools to run it locally - see below.
|
|
|
|
`llama.cpp` requires the model to be stored in the [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) file format. Models in other data formats can be converted to GGUF using the `convert_*.py` Python scripts in this repo.
|
|
To learn more about model quantization, [read this documentation](../tools/quantize/README.md)
|
|
|
|
The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with `llama.cpp`:
|
|
|
|
- Use the [GGUF-my-repo space](https://huggingface.co/spaces/ggml-org/gguf-my-repo) to convert to GGUF format and quantize model weights to smaller sizes
|
|
- Use the [GGUF-my-LoRA space](https://huggingface.co/spaces/ggml-org/gguf-my-lora) to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123)
|
|
- Use the [GGUF-editor space](https://huggingface.co/spaces/CISCai/gguf-editor) to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268)
|
|
- Use the [Inference Endpoints](https://ui.endpoints.huggingface.co/) to directly host `llama.cpp` in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669)
|