Summary
The Local AI course lab: from download to first token with benchmarks.
Key Ideas
- Running a model locally is the shortest path to understanding cost and latency.
- Two runtimes at two quantizations = the honest comparison matrix.
Lab
Choose one open model in the 7B-14B range (Qwen 2.5 7B, Llama 3.2 8B, Mistral Nemo 12B). Serve it locally with two different runtimes (e.g., Ollama and vLLM, or llama.cpp and Ollama). Benchmark both: tokens per second for decode, time to first token, memory footprint at 4-bit and 8-bit. Publish a comparison report.
Deliverables
- — Two working local deployments of the same model.
- — Benchmark script covering decode speed, TTFT, memory.
- — Report comparing the two runtimes at two quantization levels.
- — Recommendation with justification for which to use in which scenario.
Rubric
- — Correctness — both deployments serve the same model and produce compatible outputs.
- — Measurement — benchmarks are reproducible; hardware, versions, and quantization are documented.
- — Analysis — the comparison identifies specific tradeoffs, not just numbers.
- — Recommendation — the guidance is actionable, with a decision rule.