Hardware and GPUs

VRAM, bandwidth, and the constraints that matter.

Summary

The three numbers that determine what you can run locally: VRAM (holds the model), memory bandwidth (feeds the compute), and compute (does the multiplications). VRAM is the binding constraint; bandwidth is the practical bottleneck; compute matters less than most people think.

Objectives

01State the VRAM cost formula.
02Explain why bandwidth, not compute, bottlenecks inference.
03Match model sizes to consumer GPUs.

The Lesson

VRAM

Model VRAM ≈ params × bytes-per-param. Llama 70B in fp16: 140 GB. In 4-bit: 35 GB. Add ~2-10 GB for KV cache. Consumer max: 24 GB (RTX 4090, 5090). Datacenter: 80 GB (H100), 141 GB (H200, MI300X). Doesn't fit? Won't run.

Bandwidth

Inference is memory-bandwidth-bound. RTX 4090 ~1 TB/s. H100 ~3 TB/s. M2 Ultra ~800 GB/s. Higher = faster tokens-per-second. FLOPS matter for training; bandwidth matters for inference. This is why Apple Silicon punches above its FLOPS weight for LLM inference.

Key Ideas

VRAM is the yes/no; bandwidth is the how-fast.
Inference is memory-bound; compute headroom is usually wasted.

← When to run local Consumer vs. datacenter GPUs →