The three dominant quantization formats. GGUF (llama.cpp) is the ecosystem winner for CPU and Apple Silicon inference. AWQ (activation-aware) and GPTQ are optimized for GPU inference with vLLM, ExLlama, and similar. All three are real; the right choice depends on your runtime.
- 01Match each format to its runtime.
- 02State the tradeoffs among the three.
- 03Recognize the file extension of each.
GGUF
The llama.cpp format. Runs on CPU, GPU (CUDA/Metal/Vulkan), and Apple Silicon. Wide quantization variants (Q2_K through Q8_0, IQ variants). The de facto standard for local inference. Ollama, LM Studio, jan.ai all use it.
AWQ and GPTQ
Activation-aware Weight Quantization: preserves salient weights better; slight quality edge at 4-bit. GPTQ: older, still widely supported. Both are GPU-first. vLLM and TGI serve AWQ and GPTQ; llama.cpp does not natively. Choose based on your inference server, not the format's marketing.
- GGUF for local (llama.cpp ecosystem); AWQ/GPTQ for GPU serving.
- The runtime determines the format, not the other way around.