Summary
Ollama is the easiest way into local inference. One-command install, model catalog, OpenAI-compatible API, native support for CPU, CUDA, Metal. Not the fastest for high-throughput serving, but by far the fastest to first-token running.
Objectives
- 01Install Ollama and run a first model.
- 02Query Ollama with an OpenAI client.
- 03State when to graduate to vLLM.
The Lesson
Getting started
curl -fsSL https://ollama.com/install.sh | sh; ollama pull llama3.3; ollama run llama3.3. Serves on localhost:11434 with an OpenAI-compatible endpoint. Any OpenAI client library works by pointing at that base URL.
When to move on
Ollama runs one request at a time by default. For multi-user serving, batched inference, or high throughput, move to vLLM or TGI. For solo development, prototyping, and small teams, Ollama is usually enough.
Key Ideas
- Ollama is the fastest path to local inference.
- Move to vLLM when you need concurrent requests and batching.