Token bucket rate limiting
Tokens refill, requests consume, overflow rejected.
Rate limiting keeps your API from drowning in traffic—but the algorithm matters. Token bucket is the gold standard: a bucket holds tokens, requests consume them, and tokens refill at a constant rate. The beauty is its simplicity: no need to count exact timestamps, and bursts are handled gracefully by the bucket's capacity.
This diagram shows the moment-to-moment dance: a request arrives, you check the bucket, deduct a token if one exists, and accept or reject the request. The refill happens in the background at a steady pace (100 tokens per second, 1000 per minute, whatever your SLA dictates). Simple, fair, and — unlike fixed-window counters — impossible to game with a well-timed traffic burst right at the window edge.
When to use this template
- API rate-limit design review — show your team how a token bucket behaves under burst traffic and how capacity plus refill rate together control the SLA.
- Explaining rate limiting to stakeholders — product managers and partners often ask "why did my request get rejected?" This diagram is the answer, and it explains why burst traffic is OK as long as average load stays within quota.
- Implementation checklist — document what your backend does on every request: check tokens, refill if time has passed, deduct, accept-or-reject. A shared diagram keeps every language's implementation aligned.
How to adapt it
Customize the rates and capacity to your SLA:
- Change the refill rate (e.g., "10 tokens/sec" vs. "1000 tokens/minute") to reflect your API's sustainable throughput.
- Adjust the bucket capacity to control how much burst traffic you permit before throttling kicks in.
- Add a secondary bucket branch for priority or VIP clients who get a separate, higher-capacity bucket to the same API.
Visual edits regenerate clean Mermaid code, so changes to capacity and refill logic stay in version control as a reviewable, testable policy.
Mermaid code
Copy it anywhere Mermaid is supported — GitHub, Notion, or your docs.
sequenceDiagram
actor Client
participant Bucket as Token Bucket
participant API as Rate Limiter
Client->>API: Request arrives
API->>Bucket: Check token balance
alt Tokens available
Bucket->>API: Deduct 1 token, balance OK
API->>Client: Accept request
API->>Client: Process request
else Bucket empty
Bucket->>API: No tokens left
API->>Client: 429 Too Many Requests
end
Note over Bucket: Tokens refill<br/>at constant rate<br/>(e.g., 100/sec)
Frequently asked questions
- What is a token bucket algorithm?
- A rate limiter that refills a fixed-size bucket with tokens at a constant rate (e.g., 100 tokens per second). Every incoming request consumes one token. If tokens remain, the request goes through; if empty, the request is rejected with a 429. The bucket holds a maximum capacity, so tokens do not accumulate infinitely.
- How does a token bucket differ from a sliding window counter?
- Token bucket refills at a steady rate and tracks available capacity, making it smooth and bursty-friendly (a batch of requests can go through if tokens have accumulated). Sliding window tracks exact request timestamps in the past N seconds. Token bucket is simpler to implement; sliding window is more precise but needs timestamp storage.
- Why is burst traffic allowed in token bucket?
- If your bucket capacity is 100 tokens and you refill at 10 per second, a user who makes zero requests for 10 seconds will have 100 tokens stored up. Their next 100 requests will all succeed instantly. This is often desired — it permits normal traffic bursts while still bounding long-term rate. If you need to strictly reject bursts, lower the bucket capacity.
- How do I implement token bucket for a microservice API?
- Initialize a bucket with capacity and refill rate at startup. On each request, calculate tokens added since last refill, update the bucket, then check if tokens remain for this request. Many libraries (Python ratelimit, Go rate, Node rate-limiter-flexible) implement this for you. Visual edits regenerate clean Mermaid code so you can annotate your exact strategy.