Resilience & Retrying Failed Operations

Use Case: Resilience & Retrying Failed Operations

The everyday problem: Your service depends on things that fail — a database has a momentary blip, a third-party API times out, a network packet drops, a downstream service is briefly overloaded. In a distributed system, something is always failing somewhere. If your code treats every failure as fatal, a 50-millisecond network hiccup becomes a failed user request; if it retries blindly, a struggling downstream gets hammered into a full outage by a retry storm; if it waits forever on a dead dependency, threads pile up until your whole service collapses. The challenge is staying responsive and correct when dependencies misbehave — retrying what is worth retrying, giving up fast on what is not, and protecting both yourself and your dependencies from cascading failure.

Five patterns, which compose together into a resilience strategy. In .NET 8+, Microsoft.Extensions.Resilience (built on Polly v8) provides all of these as configurable pipeline strategies — so these are framework-supported, not hand-rolled.

---

Way 1: Retry with Exponential Backoff and Jitter

Best used for: Transient failures that are likely to succeed on a second attempt — a brief network glitch, a momentary database deadlock, a downstream returning 503 under temporary load. The first and most fundamental resilience pattern.

The concept: When an operation fails transiently, retry it — but not immediately and not forever. Wait progressively longer between attempts (exponential backoff: 1s, 2s, 4s) to give the dependency time to recover, and add randomness (jitter) so that many clients retrying simultaneously do not all retry at the exact same instant and re-create the overload.

// Polly v8 / Microsoft.Extensions.Resilience — retry strategy
builder.Services.AddHttpClient<IPartnerClient, PartnerClient>()
    .AddResilienceHandler("partner", pipeline =>
    {
        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            BackoffType      = DelayBackoffType.Exponential, // 2s, 4s, 8s...
            UseJitter        = true,    // randomise so clients don't retry in lockstep
            Delay            = TimeSpan.FromSeconds(2),
            // Retry ONLY transient failures — never retry a 400 Bad Request
            ShouldHandle = args => ValueTask.FromResult(
                args.Outcome.Result?.StatusCode is
                    HttpStatusCode.RequestTimeout or
                    HttpStatusCode.ServiceUnavailable or
                    HttpStatusCode.TooManyRequests
                || args.Outcome.Exception is HttpRequestException or TimeoutException)
        });
    });

// Standalone pipeline (not just HTTP) — e.g. a database operation
private static readonly ResiliencePipeline _dbRetry = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions
    {
        MaxRetryAttempts = 3,
        BackoffType      = DelayBackoffType.Exponential,
        UseJitter        = true,
        ShouldHandle = new PredicateBuilder()
            .Handle<SqlException>(ex => IsTransient(ex)) // only transient SQL errors
            .Handle<TimeoutException>()
    })
    .Build();

public Task<Order> GetOrderAsync(int id, CancellationToken ct)
    => _dbRetry.ExecuteAsync(async token => await _db.Orders.FindAsync([id], token), ct).AsTask();

Why it works: Most transient failures clear within seconds, so a few backed-off retries turn what would have been a user-facing error into an invisible recovery. Exponential backoff stops retries from piling onto a struggling dependency; jitter prevents the "retry storm" where thousands of clients synchronise their retries and repeatedly spike the dependency at the same moments.

Benchmark note: Retries dramatically reduce error rates for transient faults — a dependency with a 1% transient failure rate drops to roughly 0.0001% user-visible failures after 3 retries (assuming independence). Two rules are non-negotiable. First, only retry idempotent operations or pair retries with idempotency keys — retrying a non-idempotent POST (a payment) can charge twice (see the idempotency use case). Second, only retry transient failures — retrying a 400 Bad Request or 404 is pointless (the input is wrong, it will fail identically) and retrying 401 can lock accounts. Jitter is not optional at scale: without it, synchronised retries are a documented cause of repeated outages.

---

Way 2: Circuit Breaker (stop hammering a dead dependency)

Best used for: Protecting both your service and a failing dependency when that dependency is down (not just blipping) — when retrying would only add load to something that cannot respond and would tie up your threads waiting.

The concept: A circuit breaker tracks the failure rate of calls to a dependency. When failures exceed a threshold, it "opens" — and for a cooldown period, calls fail instantly without even attempting the dependency. After the cooldown, it goes "half-open" and lets a trial call through; if that succeeds, it closes and resumes normal operation, otherwise it re-opens. It stops you from sending doomed requests to a dead service.

pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
    FailureRatio     = 0.5,                       // open if 50%+ of calls fail...
    MinimumThroughput = 10,                       // ...over at least 10 calls (avoid tiny samples)
    SamplingDuration = TimeSpan.FromSeconds(30),  // measured across a 30s rolling window
    BreakDuration    = TimeSpan.FromSeconds(15),  // stay open 15s, then test with one call
    OnOpened   = args => { _logger.LogWarning("Circuit OPENED for partner API"); return default; },
    OnClosed   = args => { _logger.LogInformation("Circuit closed — partner recovered"); return default; },
    OnHalfOpened = args => { _logger.LogInformation("Circuit half-open — testing partner"); return default; }
});

// When open, calls throw BrokenCircuitException immediately — handle it as "dependency down"
try
{
    return await _partnerClient.GetDataAsync(ct);
}
catch (BrokenCircuitException)
{
    return CachedOrDefaultData(); // fail fast to a fallback instead of hanging
}

Why it works: When a dependency is genuinely down, every call to it is doomed and merely wastes time (waiting for timeouts) and adds load (kicking something that is already down). The open circuit converts slow, doomed calls into instant failures, freeing your threads and giving the dependency breathing room to recover. The half-open trial automatically detects recovery without you intervening.

Benchmark note: The circuit breaker is what prevents cascading failure — the scenario where a slow downstream causes upstream requests to pile up waiting, exhausting the upstream's thread pool, which makes it unresponsive, which takes down its callers, and so on until the whole system is down. Without a breaker, one slow dependency can collapse an entire service mesh. The tuning balance: too sensitive (low threshold) and it opens on normal transient noise, hurting availability; too lenient and it does not protect you. Start around a 50% failure ratio over a meaningful sample (≥10 calls) so a couple of random failures do not trip it. Retry and circuit breaker compose: retry handles brief blips, the breaker handles sustained outages.

---

Way 3: Timeout (never wait forever)

Best used for: Every outbound call, always. A call with no timeout can hang indefinitely if the dependency stops responding, and a thread waiting forever is a leaked resource that, multiplied, exhausts your capacity.

The concept: Impose a maximum time a call is allowed to take. If it exceeds that, cancel it and treat it as a failure (which retry and circuit breaker can then act on). This bounds resource consumption — a thread or connection is never tied up longer than the timeout, no matter how badly the dependency misbehaves.

// Pipeline timeout — cancels the operation if it exceeds the limit
pipeline.AddTimeout(TimeSpan.FromSeconds(5));

// Order matters in a pipeline. A robust composition (outer → inner):
//   Retry  →  Circuit Breaker  →  Timeout (per attempt)
// Each individual attempt is bounded by the timeout; retry wraps the lot.
var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(retryOptions)             // outermost: retries the whole thing
    .AddCircuitBreaker(breakerOptions)  // middle: tracks failures, opens if down
    .AddTimeout(TimeSpan.FromSeconds(5))// innermost: bounds each individual attempt
    .Build();

// Always also pass a CancellationToken so the caller (or request abort) can cancel:
public async Task<Data> GetAsync(CancellationToken ct)
    => await pipeline.ExecuteAsync(async token => await _client.FetchAsync(token), ct);

Why it works: A timeout guarantees that a resource (thread, connection, the user's patience) is released within a bounded time regardless of how the dependency fails. It converts the worst failure mode — an indefinite hang — into a clean, fast, actionable failure that the rest of your resilience pipeline can handle.

Benchmark note: The most dangerous failure is not an error — it is a call that never returns. One hung call holds a thread; under load, hung calls accumulate until the thread pool is exhausted and the service stops accepting any requests, appearing "down" even though nothing crashed. Timeouts prevent this entirely. Set them deliberately per dependency based on its real latency profile (a fast cache might time out at 1s; a slow report API at 30s), not a single global value. Crucially, ensure the CancellationToken actually flows into the underlying call so the timeout genuinely cancels the work rather than just abandoning a still-running operation — abandoned-but-running work still consumes the resource.

---

Way 4: Fallback & Graceful Degradation

Best used for: Maintaining partial service when a non-critical dependency fails — showing cached/default data instead of an error, hiding an optional feature, or returning a degraded-but-useful response rather than failing the whole request.

The concept: When an operation fails (after retries, or because the circuit is open), instead of propagating the error, substitute a sensible alternative — stale cached data, a default value, an empty result, or a simplified response. The user gets something useful, and a failure in a non-essential dependency does not break the core experience.

pipeline.AddFallback(new FallbackStrategyOptions<Recommendations>
{
    FallbackAction = args => Outcome.FromResultAsValueTask(Recommendations.Empty), // safe default
    OnFallback = args => { _logger.LogWarning("Recommendations unavailable; degrading"); return default; }
});

// Manual graceful degradation — core works even if an optional dependency is down
public async Task<ProductPage> GetProductPageAsync(int id, CancellationToken ct)
{
    var product = await _products.GetAsync(id, ct); // CRITICAL — must succeed

    // Recommendations are NICE-TO-HAVE — never let them break the page
    List<Product> recs;
    try { recs = await _recommendations.GetAsync(id, ct); }
    catch (Exception ex)
    {
        _logger.LogWarning(ex, "Recommendations failed; showing page without them");
        recs = []; // degrade gracefully — page still renders
    }

    return new ProductPage(product, recs);
}

Why it works: Not all dependencies are equally important. A recommendations widget failing should never prevent a user from seeing a product. Fallback explicitly separates critical from optional: critical failures propagate, optional ones degrade to a sensible default. The result is a system that bends instead of breaking — partially degraded beats completely down.

Benchmark note: Graceful degradation is the difference between "the whole site is down" and "one widget is missing" during a partial outage — a vast difference in user impact and on-call urgency. The design discipline it requires is classifying each dependency as critical or optional up front, and deciding the fallback for each optional one (stale cache? empty? hidden?). Serving stale cached data as a fallback is especially powerful — pair it with the cache (keep a longer-lived "stale" copy) so that when the live source fails, you serve slightly-old-but-useful data instead of an error. The only caveat: make degradation visible in monitoring (log/metric every fallback) so a silently-degrading system does not hide a real, ongoing problem.

---

Way 5: Bulkhead Isolation & Dead-Letter Handling

Best used for: Preventing one misbehaving dependency or workload from consuming all shared resources and starving everything else (bulkhead), and safely handling messages/operations that repeatedly fail so they do not block the line forever (dead-letter).

The concept: Bulkhead (named after a ship's watertight compartments) isolates resources per dependency so a flood to one cannot drain the pool the others need — e.g. cap concurrent calls to the slow partner API at 10 so they cannot consume all your threads and starve calls to the fast, critical database. Dead-letter handling deals with the operations that fail every retry: rather than retrying forever (blocking the queue) or dropping silently (losing data), route them to a dead-letter queue for inspection and later reprocessing.

// BULKHEAD — isolate concurrency per dependency so one cannot starve the others
pipeline.AddConcurrencyLimiter(permitLimit: 10, queueLimit: 5);
// At most 10 concurrent calls to THIS dependency. An 11th-onward waits (up to 5),
// then is rejected — protecting the shared thread pool from this dependency's slowness.

// In code, a SemaphoreSlim per dependency gives the same isolation:
public class IsolatedClients
{
    private readonly SemaphoreSlim _partnerBulkhead = new(10); // partner API: max 10 concurrent
    private readonly SemaphoreSlim _reportBulkhead  = new(3);  // slow reports: max 3 concurrent
    // A storm of slow report calls can occupy at most 3 slots — partner & DB calls unaffected.
}

// DEAD-LETTER — after N failed attempts, set the message aside instead of looping forever
public async Task ProcessWithDeadLetterAsync(QueueMessage msg, CancellationToken ct)
{
    try
    {
        await _processor.HandleAsync(msg, ct);
        await msg.AckAsync(ct);
    }
    catch (Exception ex)
    {
        if (msg.DeliveryCount >= 5) // tried 5 times, still failing — it's poison
        {
            _logger.LogError(ex, "Message {Id} poisoned after {N} attempts; dead-lettering",
                msg.Id, msg.DeliveryCount);
            await _deadLetterQueue.SendAsync(msg, ct); // park it for human inspection / replay
            await msg.AckAsync(ct);                    // remove from the main queue — unblock the line
        }
        else
        {
            await msg.NackAsync(ct); // let it retry (with backoff) a few more times
        }
    }
}

Why it works: Bulkheads ensure a failure or slowdown is contained — a flood to one dependency exhausts only that dependency's allotted slots, leaving capacity for everything else, so one bad actor cannot sink the whole ship. Dead-lettering ensures a permanently-failing "poison" message does not block the entire queue behind it (head-of-line blocking) or get silently lost — it is set aside, the line keeps moving, and the failed item is preserved for investigation and reprocessing once fixed.

Benchmark note: Bulkheading is what stops the classic "one slow dependency exhausted all threads and took down the whole service" incident — without it, a shared thread pool is a single point of failure that any one dependency can drain. The cost is some unused capacity reserved per compartment (you cannot use the partner API's idle slots for database calls), which is the deliberate price of isolation. Dead-letter queues are essential for any message-driven system: a single un-processable message (malformed payload, a bug in handling) without dead-lettering will be retried forever, blocking everything behind it — a common and avoidable production stall. Always monitor the dead-letter queue; a growing DLQ is an early warning of a systemic problem.

---

Choosing the right approach (and how they compose)

• Brief transient blips → Retry with backoff + jitter (Way 1)
• Sustained dependency outage → Circuit breaker (Way 2)
• Every outbound call, without exception → Timeout (Way 3)
• Non-critical dependency fails → Fallback / graceful degradation (Way 4)
• Isolate resources / handle poison messages → Bulkhead + dead-letter (Way 5)

These are not alternatives — they compose into one strategy. A robust outbound call looks like: a timeout bounds each attempt; retry handles transient failures of those attempts; a circuit breaker trips if failures are sustained so you stop hammering a dead dependency; a fallback provides a degraded response when all else fails; and bulkheads ensure this dependency's troubles cannot starve the others. In .NET 8+, AddStandardResilienceHandler() wires up a sensible default combination of these in one line, and you tune individual strategies as needed. The unifying principle: in a distributed system, failure is not exceptional — it is constant and expected, so resilience is not an add-on for the unhappy path, it is the design. And remember the prerequisite that makes safe retrying possible at all: operations must be idempotent, or retries cause duplicate side-effects.

Note: this is an informational engineering overview. Thresholds, timeouts, and failure-rate figures are illustrative starting points that must be tuned per dependency. Always validate resilience behaviour with failure injection against your own systems.