mirror of
https://github.com/ollama/ollama.git
synced 2026-09-21 13:38:14 -05:00
Jesse Gross
2e036e7cdf
The MLX runner is the only Go inference runner left and is no longer experimental, so its packages leave x/. The bindings become a top-level mlx package beside the carried patches in mlx/compat, mirroring how llama/ holds the llama.cpp integration, and the runner becomes mlxrunner with the architectures nested under the package they implement. Subpackages move with their parent unless listed. x/mlxrunner/mlx mlx x/internal/mlxthread mlx/mlxthread x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest x/internal/mlxtest mlx/mlxtest x/quant mlx/quant mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx) x/mlxrunner mlxrunner x/models/nn mlxrunner/nn x/models/<arch> mlxrunner/model/<arch> x/mlxrunner/imports.go mlxrunner/model/architectures (new package) x/create create x/safetensors fs/safetensors x/tokenizer mlxrunner/tokenizer Every package keeps its name, so the Go changes are the import path rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift check and Darwin payload script follow the new paths. Four edits are not paths: the runner's blank architecture imports become the package mlxrunner/model/architectures, so the list to extend for a new model sits beside the architecture directories; a depguard rule keeps the two test harnesses out of non-test code, as the x/internal placement used to; the CI change filter's two entries for the long-deleted x/imagegen/mlx now name the bindings' CMake project and the carried patches, so a change to either builds the payload; and the tokenizer parity test reads its fixtures from its own testdata instead of walking out of x/. x/server and x/imagegen/manifest stay for the next two commits.
99 lines
3.9 KiB
Markdown
99 lines
3.9 KiB
Markdown
# Llama
|
|
|
|
## Updating llama.cpp
|
|
|
|
`LLAMA_CPP_VERSION` pins Ollama's llama.cpp source. An update can change more
|
|
than compilation: it can affect model loading, GPU discovery, scheduler inputs,
|
|
runtime logs, streaming, and compatibility patches. Validate the upstream diff,
|
|
the patched source Ollama actually builds, and the affected local paths.
|
|
|
|
### Workflow
|
|
|
|
Record the old ref from the base branch and choose an explicit new llama.cpp
|
|
tag or commit. After updating `LLAMA_CPP_VERSION`, materialize the source
|
|
through Ollama's normal build path:
|
|
|
|
```sh
|
|
cmake -S llama/server --preset cpu
|
|
```
|
|
|
|
This configure step fetches the pinned source and applies `llama/compat/`
|
|
patches. Confirm the resulting checkout, usually
|
|
`build/llama-server-cpu/_deps/llama_cpp-src`, resolves to the intended new ref.
|
|
Do not trust an old or dirty `_deps/` checkout as validation.
|
|
This is only a source and patch-application check; it is not runtime
|
|
validation.
|
|
|
|
Review the upstream diff using Git refs from the llama.cpp checkout:
|
|
|
|
```sh
|
|
git diff <old-ref> <new-ref> -- <path>
|
|
git show <new-ref>:<path>
|
|
```
|
|
|
|
Avoid treating patched working-tree files as pristine upstream source.
|
|
|
|
For build prerequisites, platform notes, and backend selection, see the
|
|
[developer guide](../docs/development.md).
|
|
|
|
### What to review
|
|
|
|
- Build option and dependency drift: changed `GGML_*` or `LLAMA_*` options,
|
|
new `find_package` calls, generated assets, shader tools, or backend
|
|
dependencies. Compare against `llama/server/CMakeLists.txt`,
|
|
`llama/server/CMakePresets.json`, `cmake/local.cmake`, Dockerfiles, CI, and
|
|
build scripts as needed.
|
|
- Backend discovery contracts: GGML symbols used by `discover/native_probe*.go`,
|
|
`ggml_backend_dev_props`, backend device type enums, backend registry loading,
|
|
device ordering, visible-device filtering, and CUDA/ROCm/Vulkan/Metal runtime
|
|
library behavior.
|
|
- llama-server contracts: launch args and defaults, status and error payloads,
|
|
memory/offload log lines, `system_info:`, flash-attention logging,
|
|
`--main-gpu`, split-mode behavior, and scheduler-sensitive flags consumed by
|
|
`llm/llama_server.go` or `server/sched.go`.
|
|
- Streaming: any new SSE frame shape, heartbeat, keepalive ping, completion
|
|
marker, or response cadence on paths Ollama parses directly.
|
|
- Model and conversion surfaces: new architectures, tensor names, GGUF
|
|
metadata, tokenizer behavior, speculative/MTP paths, sampler defaults, and
|
|
server capabilities that may require updates under `model/`,
|
|
`create/`, `llm/`, or `llama/compat/`. A model load alone is not enough;
|
|
affected paths should run a real request and assert the expected result.
|
|
|
|
### Compatibility patches
|
|
|
|
Patches under `llama/compat/` are applied during configure. If a patch
|
|
insertion point moved, regenerate the patch against a fresh checkout of the new
|
|
ref rather than editing an already-patched `_deps/` tree.
|
|
|
|
If compatibility sources, model patches, `llama/server/CMakeLists.txt`, or
|
|
`cmake/local.cmake` changed, build the CPU target:
|
|
|
|
```sh
|
|
cmake --build build/llama-server-cpu --target llama-server --parallel 12
|
|
```
|
|
|
|
Configure-only validation can miss missing sources, template instantiation
|
|
problems, and link errors. Also check whether upstream now supports a locally
|
|
patched model natively; if it does, the local patch may need removal or rebase.
|
|
|
|
### Local checks
|
|
|
|
Run the Go tests:
|
|
|
|
```sh
|
|
go test ./...
|
|
```
|
|
Then proceed to build the full Ollama release and verify.
|
|
|
|
### End-to-end Testing
|
|
|
|
For runtime validation, build the full applicable native payload for the
|
|
platform using the [developer guide](../docs/development.md): Metal on macOS
|
|
arm64, and the available CUDA, ROCm, and Vulkan backends on Linux and Windows.
|
|
|
|
Then run the [integration tests](../integration/README.md) on the platforms
|
|
being validated. Use them to exercise real Ollama requests and inspect logs for
|
|
device discovery, offload, memory accounting, flash attention, and
|
|
request/response behavior. macOS, Windows, and Linux behavior must be validated
|
|
on those platforms.
|