Jesse Gross
2e036e7cdf
The MLX runner is the only Go inference runner left and is no longer experimental, so its packages leave x/. The bindings become a top-level mlx package beside the carried patches in mlx/compat, mirroring how llama/ holds the llama.cpp integration, and the runner becomes mlxrunner with the architectures nested under the package they implement. Subpackages move with their parent unless listed. x/mlxrunner/mlx mlx x/internal/mlxthread mlx/mlxthread x/internal/mlxthreadtest mlx/mlxthread/mlxthreadtest x/internal/mlxtest mlx/mlxtest x/quant mlx/quant mlx/compat/*.patch mlx/compat/mlx-c (MLX patches go in mlx/compat/mlx) x/mlxrunner mlxrunner x/models/nn mlxrunner/nn x/models/<arch> mlxrunner/model/<arch> x/mlxrunner/imports.go mlxrunner/model/architectures (new package) x/create create x/safetensors fs/safetensors x/tokenizer mlxrunner/tokenizer Every package keeps its name, so the Go changes are the import path rewrites the moves force, and the CMake, Dockerfile, CI cache keys, drift check and Darwin payload script follow the new paths. Four edits are not paths: the runner's blank architecture imports become the package mlxrunner/model/architectures, so the list to extend for a new model sits beside the architecture directories; a depguard rule keeps the two test harnesses out of non-test code, as the x/internal placement used to; the CI change filter's two entries for the long-deleted x/imagegen/mlx now name the bindings' CMake project and the carried patches, so a change to either builds the payload; and the tokenizer parity test reads its fixtures from its own testdata instead of walking out of x/. x/server and x/imagegen/manifest stay for the next two commits.
Llama
Updating llama.cpp
LLAMA_CPP_VERSION pins Ollama's llama.cpp source. An update can change more
than compilation: it can affect model loading, GPU discovery, scheduler inputs,
runtime logs, streaming, and compatibility patches. Validate the upstream diff,
the patched source Ollama actually builds, and the affected local paths.
Workflow
Record the old ref from the base branch and choose an explicit new llama.cpp
tag or commit. After updating LLAMA_CPP_VERSION, materialize the source
through Ollama's normal build path:
cmake -S llama/server --preset cpu
This configure step fetches the pinned source and applies llama/compat/
patches. Confirm the resulting checkout, usually
build/llama-server-cpu/_deps/llama_cpp-src, resolves to the intended new ref.
Do not trust an old or dirty _deps/ checkout as validation.
This is only a source and patch-application check; it is not runtime
validation.
Review the upstream diff using Git refs from the llama.cpp checkout:
git diff <old-ref> <new-ref> -- <path>
git show <new-ref>:<path>
Avoid treating patched working-tree files as pristine upstream source.
For build prerequisites, platform notes, and backend selection, see the developer guide.
What to review
- Build option and dependency drift: changed
GGML_*orLLAMA_*options, newfind_packagecalls, generated assets, shader tools, or backend dependencies. Compare againstllama/server/CMakeLists.txt,llama/server/CMakePresets.json,cmake/local.cmake, Dockerfiles, CI, and build scripts as needed. - Backend discovery contracts: GGML symbols used by
discover/native_probe*.go,ggml_backend_dev_props, backend device type enums, backend registry loading, device ordering, visible-device filtering, and CUDA/ROCm/Vulkan/Metal runtime library behavior. - llama-server contracts: launch args and defaults, status and error payloads,
memory/offload log lines,
system_info:, flash-attention logging,--main-gpu, split-mode behavior, and scheduler-sensitive flags consumed byllm/llama_server.goorserver/sched.go. - Streaming: any new SSE frame shape, heartbeat, keepalive ping, completion marker, or response cadence on paths Ollama parses directly.
- Model and conversion surfaces: new architectures, tensor names, GGUF
metadata, tokenizer behavior, speculative/MTP paths, sampler defaults, and
server capabilities that may require updates under
model/,create/,llm/, orllama/compat/. A model load alone is not enough; affected paths should run a real request and assert the expected result.
Compatibility patches
Patches under llama/compat/ are applied during configure. If a patch
insertion point moved, regenerate the patch against a fresh checkout of the new
ref rather than editing an already-patched _deps/ tree.
If compatibility sources, model patches, llama/server/CMakeLists.txt, or
cmake/local.cmake changed, build the CPU target:
cmake --build build/llama-server-cpu --target llama-server --parallel 12
Configure-only validation can miss missing sources, template instantiation problems, and link errors. Also check whether upstream now supports a locally patched model natively; if it does, the local patch may need removal or rebase.
Local checks
Run the Go tests:
go test ./...
Then proceed to build the full Ollama release and verify.
End-to-end Testing
For runtime validation, build the full applicable native payload for the platform using the developer guide: Metal on macOS arm64, and the available CUDA, ROCm, and Vulkan backends on Linux and Windows.
Then run the integration tests on the platforms being validated. Use them to exercise real Ollama requests and inspect logs for device discovery, offload, memory accounting, flash attention, and request/response behavior. macOS, Windows, and Linux behavior must be validated on those platforms.