Rate Limiting & Throttling

Use Case: Rate Limiting & Throttling

The everyday problem: One client — a buggy script, an aggressive scraper, a misconfigured integration, or an outright attacker — hammers your API with thousands of requests per second. Your database connection pool exhausts, response times for every other user collapse, and the whole service degrades because of one bad actor. Or your free tier needs to allow 100 requests/day while paid tiers get 10,000. Or a downstream third-party API charges per call and you must not exceed their quota. The challenge is controlling how many requests a client can make in a given window — protecting your service from overload and abuse while treating well-behaved clients fairly.

Five algorithms/approaches, from the simplest counter to distributed limiting across a server farm. .NET 7+ ships a built-in RateLimiter in System.Threading.RateLimiting and rate-limiting middleware, so most of these are now first-class framework features.

---

Way 1: Fixed Window

Best used for: Simple per-client request caps where occasional bursts at window boundaries are acceptable — basic API quotas, "100 requests per minute" tiers. The easiest to understand and implement.

The concept: Divide time into fixed windows (e.g. each calendar minute). Count requests per client in the current window. When the count hits the limit, reject further requests until the window resets. At the boundary, the counter snaps back to zero.

// .NET built-in fixed-window limiter
// Program.cs
builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter("fixed", opt =>
    {
        opt.Window            = TimeSpan.FromMinutes(1);
        opt.PermitLimit       = 100;          // 100 requests per minute per partition
        opt.QueueLimit        = 0;            // reject immediately when over limit
        opt.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
    });

    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.OnRejected = async (context, ct) =>
    {
        context.HttpContext.Response.Headers.RetryAfter = "60";
        await context.HttpContext.Response.WriteAsync("Rate limit exceeded. Retry in 60s.", ct);
    };
});

var app = builder.Build();
app.UseRateLimiter();

// Apply per-endpoint
app.MapGet("/api/data", () => GetData()).RequireRateLimiting("fixed");

// Per-client partitioning — limit each API key / user / IP separately
builder.Services.AddRateLimiter(options =>
{
    options.AddPolicy("per-client", httpContext =>
        RateLimitPartition.GetFixedWindowLimiter(
            // Partition key: the thing you limit BY (API key, user id, or IP)
            partitionKey: httpContext.Request.Headers["X-Api-Key"].ToString()
                          ?? httpContext.Connection.RemoteIpAddress?.ToString()
                          ?? "anonymous",
            factory: _ => new FixedWindowRateLimiterOptions
            {
                Window      = TimeSpan.FromMinutes(1),
                PermitLimit = 100
            }));
});

Why it works: A single integer counter per client per window — trivially cheap and easy to reason about. The partition key ensures each client gets their own independent quota rather than sharing a global one.

Benchmark note: The well-known flaw is the boundary burst: a client can send 100 requests at 11:59:59 and another 100 at 12:00:00 — 200 requests in two seconds — because the counter resets at the window edge. For a 100/minute limit this means a theoretical peak of 200/minute across a boundary. If that burst can hurt you, use sliding window (Way 2). The cost of the limiter itself is negligible (an in-memory counter), so the choice is purely about burst tolerance.

---

Way 2: Sliding Window

Best used for: When the fixed-window boundary burst is unacceptable and you need smoother enforcement — APIs where the limit must genuinely mean "no more than N in any 60-second period", not "per calendar minute".

The concept: The window slides continuously rather than snapping. The framework's implementation divides the window into segments and expires them incrementally, so the count reflects a rolling period. A request at 12:00:00 still "sees" the requests from 11:59:30 in its trailing 60-second window, preventing the boundary doubling.

builder.Services.AddRateLimiter(options =>
{
    options.AddSlidingWindowLimiter("sliding", opt =>
    {
        opt.Window           = TimeSpan.FromMinutes(1);
        opt.PermitLimit      = 100;
        opt.SegmentsPerWindow = 6;     // window split into 6 × 10-second segments
        opt.QueueLimit       = 0;
    });
});
// As time advances, the oldest 10-second segment's count is subtracted and a new
// segment begins — so the effective count always covers the trailing 60 seconds.

Why it works: By tracking sub-segments and expiring them progressively, the limiter approximates a true rolling window without storing a timestamp for every individual request. More segments = smoother approximation (and slightly more memory).

Benchmark note: Sliding window eliminates the boundary-burst problem of fixed window at the cost of tracking SegmentsPerWindow counters per client instead of one — still negligible memory. With 6 segments, the granularity of expiry is 10 seconds; a true per-request sliding log would be exact but far more expensive. For nearly all APIs, segmented sliding window is the right balance: it caps the real rate without the 2× boundary loophole, and the extra cost over fixed window is immaterial.

---

Way 3: Token Bucket (allow controlled bursts)

Best used for: APIs that want to allow short legitimate bursts while capping the sustained average rate — a user who is idle then makes 20 quick requests should be allowed, but nobody should sustain 1,000/minute forever. The model most public APIs (AWS, Stripe-style) use.

The concept: Each client has a bucket holding up to N tokens. Each request consumes one token. Tokens refill at a steady rate (e.g. 10 per second). A client who has not made requests accumulates tokens up to the bucket cap, so they can burst that many at once; once drained, they are limited to the refill rate. Burst capacity and sustained rate are tuned independently.

builder.Services.AddRateLimiter(options =>
{
    options.AddTokenBucketLimiter("token", opt =>
    {
        opt.TokenLimit          = 50;    // bucket holds up to 50 tokens → max burst of 50
        opt.TokensPerPeriod     = 10;    // refill 10 tokens...
        opt.ReplenishmentPeriod = TimeSpan.FromSeconds(1); // ...every second → 10/sec sustained
        opt.QueueLimit          = 0;
        opt.AutoReplenishment   = true;
    });
});
// Idle client accumulates up to 50 tokens, can fire 50 requests instantly,
// then is throttled to 10/second as the bucket refills.

Why it works: It separates two concerns that fixed/sliding windows conflate: burst capacity (bucket size) and sustained rate (refill rate). This matches real usage — bursty-but-bounded clients are common and legitimate, and token bucket accommodates them without raising the long-run average limit.

Benchmark note: Token bucket is the most flexible single-client algorithm and is what you want when "allow reasonable bursts, cap the average" describes your need. Tuning is the work: TokenLimit too high lets clients overwhelm a fragile downstream in a burst; refill too slow frustrates legitimate bursty users. A common starting point is bucket = a few seconds' worth of sustained rate. Memory and CPU cost per client are the same order as the other algorithms — a small struct of state and a refill timer.

---

Way 4: Concurrency Limiter (cap simultaneous, not rate)

Best used for: Protecting a resource that is sensitive to simultaneous load rather than request rate — an expensive report endpoint, a call to a downstream service that allows only N concurrent connections, or a CPU-bound operation. Limits "how many at once", not "how many per minute".

The concept: Instead of counting requests over time, you cap the number of requests being processed concurrently. The 11th request waits (queued) until one of the 10 in-flight requests finishes, or is rejected if the queue is full. This is back-pressure: it matches admission to actual processing capacity.

builder.Services.AddRateLimiter(options =>
{
    options.AddConcurrencyLimiter("concurrency", opt =>
    {
        opt.PermitLimit = 10;    // at most 10 requests processing this endpoint at once
        opt.QueueLimit  = 20;    // up to 20 more may wait; beyond that → 429
        opt.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
    });
});

// Apply to an expensive endpoint
app.MapPost("/api/generate-pdf", GeneratePdf).RequireRateLimiting("concurrency");
// Protects the PDF generator from being asked to do 500 conversions simultaneously

// The same idea inside code, gating a downstream call with SemaphoreSlim
public class ThrottledDownstreamClient
{
    private readonly SemaphoreSlim _gate = new(initialCount: 5); // max 5 concurrent calls
    private readonly HttpClient _http;

    public async Task<T> CallAsync<T>(string url, CancellationToken ct)
    {
        await _gate.WaitAsync(ct);            // wait for a slot
        try { return await _http.GetFromJsonAsync<T>(url, ct); }
        finally { _gate.Release(); }          // always release the slot
    }
}

Why it works: Some resources do not care about rate — they care about simultaneity. A downstream that allows 5 concurrent connections is unaffected by 1,000 requests/minute as long as no more than 5 run at once. Concurrency limiting maps admission directly to capacity, providing natural back-pressure instead of overwhelming the resource and failing.

Benchmark note: This is the right tool when the failure mode is "too many at the same instant" rather than "too many over time". It prevents thundering-herd collapse of expensive endpoints. The tuning risk is latency: a low PermitLimit with high traffic means requests queue and overall latency rises even though nothing is rejected — monitor queue wait time. Pair concurrency limiting with a timeout so a request does not wait forever in the queue. SemaphoreSlim is the in-process primitive; the middleware version handles the HTTP queuing and rejection for you.

---

Way 5: Distributed Rate Limiting (Redis across many servers)

Best used for: Any API running on more than one instance behind a load balancer where the limit must be enforced globally — "100 requests/minute per API key" must mean across all servers combined, not 100 per server.

The concept: The built-in limiters store counts in each server's local memory, so with 5 servers a "100/minute" limit becomes effectively 500/minute (100 per server). To enforce a true global limit, the counter must live in shared storage — typically Redis — and be updated atomically. Every server checks and increments the same Redis counter, so the limit is shared.

// Distributed fixed-window limiter backed by Redis (atomic INCR + EXPIRE)
public class RedisRateLimiter
{
    private readonly IDatabase _redis;
    public RedisRateLimiter(IConnectionMultiplexer mux) => _redis = mux.GetDatabase();

    public async Task<bool> IsAllowedAsync(string clientKey, int limit, TimeSpan window)
    {
        // Window-bucketed key so it auto-rolls each period
        var bucket = DateTimeOffset.UtcNow.ToUnixTimeSeconds() / (long)window.TotalSeconds;
        var key    = $"ratelimit:{clientKey}:{bucket}";

        // Atomic: increment the shared counter; set expiry on first hit
        long count = await _redis.StringIncrementAsync(key);
        if (count == 1)
            await _redis.KeyExpireAsync(key, window);

        return count <= limit; // every server sees the SAME count → global enforcement
    }
}

// For accuracy under high concurrency, do the check-and-increment in a single
// atomic Lua script so the INCR and the limit comparison cannot interleave:
// EVAL "local c = redis.call('INCR', KEYS[1])
//       if c == 1 then redis.call('EXPIRE', KEYS[1], ARGV[2]) end
//       return c <= tonumber(ARGV[1]) and 1 or 0" 1 key limit windowSeconds

// Use it as middleware
app.Use(async (ctx, next) =>
{
    var clientKey = ctx.Request.Headers["X-Api-Key"].ToString();
    if (!await _redisLimiter.IsAllowedAsync(clientKey, limit: 100, window: TimeSpan.FromMinutes(1)))
    {
        ctx.Response.StatusCode = StatusCodes.Status429TooManyRequests;
        ctx.Response.Headers.RetryAfter = "60";
        return;
    }
    await next();
});

Why it works: A single shared counter in Redis, updated atomically, means all servers enforce one global limit regardless of which server a request lands on. Redis's atomic INCR (and Lua scripting for combined check-and-increment) handles the concurrency correctly even when many servers hit the same key simultaneously.

Benchmark note: The cost is one Redis round-trip per request (~0.2–1 ms on a local network) — usually negligible, but it does add a hard dependency: if Redis is down, you must decide fail-open (allow requests, lose enforcement) or fail-closed (reject everything). Most APIs fail-open for availability and alert on it. For very high throughput, the Lua-script approach avoids a race between the INCR and the comparison. Sliding-window and token-bucket variants exist as Redis Lua scripts too (and libraries like RedisRateLimiting wrap them into the .NET limiter abstraction). This is mandatory the moment you run more than one instance and the limit must be exact.

---

Choosing the right approach

• Simple quota, bursts at boundary OK, single server → Fixed window (Way 1)
• Need smooth enforcement, no boundary loophole → Sliding window (Way 2)
• Allow legitimate bursts, cap sustained average → Token bucket (Way 3)
• Protect against simultaneous load, not rate → Concurrency limiter (Way 4)
• Multiple servers, limit must be global → Distributed Redis (Way 5)

The decision spine: first ask rate or concurrency? — if the resource fails under simultaneous load, use a concurrency limiter (Way 4); otherwise you are limiting rate. For rate, pick the algorithm by burst tolerance: fixed window if boundary bursts are harmless, sliding window for smooth caps, token bucket when you want to permit bursts up to a ceiling. Then ask how many servers? — anything beyond a single instance needs the count in shared storage (Way 5) or your limit silently multiplies by the server count. Always return 429 Too Many Requests with a Retry-After header so well-behaved clients back off correctly, and apply limits per meaningful partition (API key or user, falling back to IP) rather than globally, so one client cannot consume everyone else's quota.

Note: this is an informational engineering overview. Latency and capacity figures are illustrative and depend on infrastructure and configuration. Always measure against your own environment.