Every provider rate-limits by requests per minute, tokens per minute, and often per-model quotas. A production client needs exponential backoff, jitter, and provider-specific queue awareness. The retry logic is the difference between a robust service and a fragile one.
- 01State the standard backoff formula.
- 02Explain the role of jitter.
- 03Name the two rate-limit types every provider enforces.
The formula
On 429 or 503: wait = min(cap, base × 2^attempt) + random jitter. Base = 1s, cap = 60s, max attempts = 5-8 depending on tolerance. Never retry without backoff; you will DoS the provider and get further rate-limited.
Provider-specific
OpenAI: rate limits per tier, visible in headers. Anthropic: token-per-minute limits; Retry-After header. Google: quota per project, per-region. Read the response headers; every provider exposes remaining quota and reset time.
Minimal retry loop
for attempt in range(8): try: return call(); except RateLimitError: sleep(min(60, 2 ** attempt) + random.uniform(0, 1))
- Exponential backoff with jitter; retry-after when the header is present.
- Rate-limit headers are the source of truth.