Summary
llama.cpp is the C++ engine that runs GGUF-quantized models on nearly any hardware. Ollama, LM Studio, Jan, and half the local-AI ecosystem sit on top of it. Understanding llama.cpp is understanding the substrate.
Objectives
- 01Build and run llama.cpp from source.
- 02State the hardware backends it supports.
- 03Name three tools that wrap it.
The Lesson
The engine
C++ library implementing transformer inference with GGUF quantization. Backends: CPU (AVX2, AVX-512, NEON), CUDA, Metal, Vulkan, ROCm, SYCL. Ships with a server binary that exposes an OpenAI-compatible API. Fast, portable, lean.
The ecosystem
Ollama, LM Studio, Jan, GPT4All, koboldcpp, text-generation-webui, LocalAI — all use llama.cpp as their inference core. The Hugging Face GGUF ecosystem exists because of it. When llama.cpp adds a feature (new quantization, new architecture), the whole ecosystem gains it.
Key Ideas
- llama.cpp is the substrate under local inference.
- Learning llama.cpp is learning why every local tool behaves the way it does.