Summary
Streaming returns tokens as they are generated instead of after the full response completes. It is the single largest UX improvement in AI applications — users see first-token latency drop from seconds to hundreds of milliseconds. Every serious chat application streams.
Objectives
- 01State the SSE format used by streaming APIs.
- 02Handle streaming in application code.
- 03Recognize when streaming is the wrong choice.
The Lesson
The mechanism
Server-Sent Events (SSE) or chunked HTTP. The provider sends 'data: {...}' lines as tokens generate; the client parses and displays incrementally. Every major provider supports it; every OpenAI-compatible client handles it.
When not to stream
Structured output where the client needs the parsed result (though even here, streaming JSON parsers exist). Background jobs where latency does not matter. Batch processing. Stream when a human is waiting; don't when a machine is.
Key Ideas
- Streaming turns 3s latency into 300ms perceived latency.
- Stream when a human waits; batch when a machine consumes.