A token is the atomic input and output unit of a language model. It is neither a character nor a word — it is a subword chunk chosen by a learned tokenizer. Every cost, every context limit, every rate limit is measured in tokens. The unit matters more than most operators realize.
- 01Define a token in one sentence.
- 02Estimate the token count of English text without a tokenizer.
- 03Explain why the same string has different token counts across providers.
The unit
GPT-4 tokenizes 'unbelievable' as roughly ['un', 'believ', 'able'] — three tokens for one word. Common English averages ~4 characters per token; code averages more tokens per line; non-Latin scripts often produce 2-4x more tokens than Latin equivalents.
Why providers differ
Each model family trains its own tokenizer on its own corpus. GPT-4o's tokenizer is different from Claude's, which is different from Gemini's. A prompt that costs 1,000 tokens on one is 1,150 on another. Budget models by measured, not estimated, token counts.
Rule of thumb
For English prose: characters ÷ 4 ≈ token count. For code: characters ÷ 3. For Japanese: characters × 1.5. Always verify with the provider's tokenizer before committing to a budget.
- Tokens are subwords, not words or characters.
- Token counts vary across providers; measure, don't estimate.