← Back to System Internals

Designing a Rate Limiter

A system design interview guide to building a rate limiter: why you need one, where it sits, the classic algorithms and their tradeoffs, and how to make it correct and fast in a distributed setting.

A rate limiter controls how many requests a client may send to a service in a given window of time. When a client stays under its allowance, requests pass through to the service; when it exceeds the allowance, the limiter rejects or defers the extra requests instead of letting them reach the backend. This sounds simple, but a good design has to answer several harder questions: which algorithm decides what "too many" means, where the limiter physically lives, how it stays accurate when many requests arrive at once, and how multiple limiter instances agree on a single count. This guide walks through each of those decisions in the order an interview usually takes them.

1. Why Rate Limit

Rate limiting is one of those features that looks like a tax on legitimate traffic until the day it saves the service. There are four distinct reasons to add one, and a strong answer names all of them rather than just "to stop abuse."

2. Where to Place the Limiter

Before choosing an algorithm, decide where the check runs. There are three common placements, and they trade enforceability against blast radius and reuse.

• Client side. The client throttles its own requests before sending them. This is cheap and saves a round trip, but it is unenforceable — anyone can modify the client or forge requests — so it is at best a courtesy, never a security control.
• Server side, inside the application. The service checks the limit as part of handling each request. This is enforceable and has full request context, but every service has to reimplement the logic, and the rejected request still pays for connection setup and routing before it is dropped.
• Middleware or an API gateway. A dedicated layer in front of the services inspects every request and applies the limit before forwarding it. This is the usual choice: it is enforceable, it is shared by all backends behind it, and it sheds excess load early, before it reaches application servers. The cost is one more component to operate and a network hop.

For most designs you choose the gateway/middleware placement. It centralizes the rules, keeps the limiting logic out of every individual service, and rejects bad traffic at the cheapest possible point in the path.

3. Algorithms and Tradeoffs

The heart of the design is the algorithm that decides whether a given request is allowed. Five are worth knowing. They differ in how they track usage, whether they permit bursts, and how much memory and accuracy they cost.

The sliding window counter is worth spelling out, because its formula is the part interviewers most often ask you to derive. The estimated request count for the current rolling window is:

# count over the rolling window =
estimate = current_window_count
         + previous_window_count * overlap_percentage_of_rolling_window

# overlap_percentage = fraction of the previous fixed window
# that still falls inside the rolling window right now
allow if estimate < limit

For example, with a one-minute window and a limit of 100: if the previous minute saw 80 requests, the current minute has 30 so far, and 25% of the previous window still overlaps the rolling window, the estimate is 30 + 80 × 0.25 = 50, which is under 100, so the request is allowed. As the rolling window advances, the previous window's weight shrinks toward zero, smoothly retiring its contribution instead of dropping it all at once like the fixed window does.

4. Architecture with Redis

Put the pieces together. A request from the client hits the rate limiter middleware. The middleware looks up the rules that apply to this client and the counters tracking its recent usage, runs the chosen algorithm, and either forwards the request to the API servers or rejects it. The counters need to be read and written on every request, so they live in a fast in-memory store; the rules change rarely, so they are cached and refreshed in the background.

The components are:

• Rate limiter middleware. Sits in front of the API servers. For each request it identifies the client (API key, user ID, or IP), fetches the relevant counter and rule, applies the algorithm, and decides allow or reject.
• Redis (counters). An in-memory key-value store holding the per-client counters. It is chosen because the limiter touches a counter on every single request: an in-memory store gives sub-millisecond reads and writes, and Redis ships with INCR and key expiration that map directly onto window counting.
• Rules cache and workers. The limiting rules (which client gets which limit) are kept in a durable rules store. Background workers periodically load them into an in-memory cache next to the middleware, so the hot path never queries the database. A rule change propagates on the next refresh.

Why an in-memory store rather than a database? Counters change on every request and must be read with very low latency, or the limiter itself becomes the bottleneck it was meant to prevent. A disk-backed database cannot keep up at that rate; Redis can, and its single-threaded command execution plus native counter and TTL primitives make it a natural fit.

The fixed-window check in Redis is two commands — increment the counter for the current window, and set it to expire when the window ends:

function allow(client_id):
  key = client_id + ":" + current_window()   # e.g. "user42:1719356400"
  count = redis.INCR(key)                     # create at 1 or bump existing
  if count == 1:
    redis.EXPIRE(key, window_seconds)         # auto-clean old windows
  return count <= limit

This naive version has a race: between INCR and EXPIRE, or when many requests read the same counter concurrently, two requests can both believe they are under the limit. The fix is to make the read-decide-write sequence atomic by running it as a single Lua script on the Redis server, which executes without interleaving:

# Lua script runs atomically on the Redis server
function allow_atomic(key, limit, window_seconds):
  count = redis.call("INCR", key)
  if count == 1:
    redis.call("EXPIRE", key, window_seconds)
  return count <= limit ? 1 : 0
# the whole check is one indivisible operation — no race

5. Responding to the Client

When a request is rejected, the limiter returns HTTP 429 Too Many Requests. The status code alone is not enough; a well-behaved client needs to know its limit and when it can try again, so the limiter attaches rate-limit headers to the response.

Returning these headers on allowed responses too — not just on rejections — lets a well-behaved client slow itself down before it hits the wall, which reduces the number of 429s the system has to generate in the first place.

6. Handling Exceeded Requests

When a request goes over the limit, you have two choices for what to do with it, and the right one depends on whether the work is interactive or deferrable.

• Drop it. Reject the request immediately with a 429 and let the client decide whether to retry. This is the right behavior for interactive, latency-sensitive traffic — a user-facing API call — where a stale, delayed response is worse than a clear "try again."
• Enqueue it for later. Instead of dropping the request, push it onto a message queue to be processed when capacity frees up. This suits asynchronous work that does not need an immediate answer — a batch job, an email send, an event ingestion — where completing the work eventually matters more than completing it now. The cost is added complexity and the need to handle queue growth under sustained overload.

In practice many systems do both: drop excess requests on synchronous, user-facing paths, and enqueue excess work on asynchronous paths where deferral is acceptable.

7. Distributed Challenges

A single limiter instance with a local counter is easy. Real systems run many limiter nodes behind a load balancer, and that introduces two problems the design must address.

• Race conditions. When several requests for the same client land on different nodes (or the same node concurrently) and all read the counter at nearly the same moment, each may see a value under the limit and allow its request, so the combined total overshoots. This is the same race described earlier, and the same fix applies: make the read-and-update atomic at the shared store, using a Lua script or transaction so concurrent updates serialize.
• Synchronization across nodes. Each node must enforce a single, shared limit, not its own private one. There are two ways to achieve that:

The centralized store is the standard answer because it keeps the limiter nodes stateless and the count consistent. The latency of the extra hop is small with an in-memory store, and the atomic-update technique handles the races that the shared counter would otherwise invite.

8. Summary

A rate limiter is a sequence of decisions, each building on the last: