noun · API concept
Rate limit
also called throttling
A speed limit on how much you can ask. Go over it and the service tells you to wait a bit and try again.
A cap on how many requests or tokens an account can use in a given time period, enforced by the API provider.
The service counts your requests and tokens over a time window and rejects calls beyond the cap, usually with an HTTP 429 status and sometimes a hint about how long to wait. Big prompts use more of the token budget, so a few large requests can hit the limit as fast as many small ones.
Real-life example
Your batch job processes a thousand documents and starts failing partway through. The service is not broken. You sent requests faster than your limit allows, and it began rejecting them. Spacing out calls and retrying after a pause, with growing delays, gets the job through.
HTTP 429 Too Many RequestsPlayground
Fire off requests faster than your allowance refills, then compare retrying instantly with backing off.
Your app
- Accepted
- 0
- Rejected (429)
- 0
- Gave up
- 0
Your allowance
6.0 of 6 left
Last 20 seconds
accepted rejected (429) Lower dots are retries.
Retrying instantly spends your allowance on calls that will fail again. Doubling the wait each time, with a little randomness, lets the bucket refill.
Illustrative: a bucket that holds 6 and refills 1 per second. Real limits differ by provider and plan, and count tokens as well as requests.
Where the analogy breaks
-
Limits count tokens as well as requests. A few huge prompts can use up your allowance as fast as many small ones.
-
Hitting one is normal and expected. Build in retries that wait longer each time, and do not hammer the service.
-
Limits differ by account and plan. Check your own, because they can change as your usage and standing change.