Summary
Ollama runs open-weight models locally with an OpenAI-compatible API. Not just a desktop client — a serious server for local, on-prem, or air-gapped deployment. Any OpenAI client library talks to Ollama by changing the base URL.
Objectives
- 01Start an Ollama server and query it.
- 02State the OpenAI-compatibility layer.
- 03Explain when local serving beats API.
The Lesson
The server
ollama serve exposes a local endpoint at localhost:11434. Set OPENAI_BASE_URL and you're using Ollama with any OpenAI client. Supports LLMs, embeddings, vision models, function calling on models that support it.
When it wins
Privacy requirements (no data leaves the network). Cost at very high volume (once GPUs are amortized). Offline or air-gapped environments. Not: latency-sensitive apps on consumer hardware, or workloads that need the strongest available model.
Key Ideas
- Ollama = local serving with OpenAI API shape.
- Wins on privacy, cost-at-volume, and air-gap; loses on peak capability.