Idempotency & Duplicate Requests

Use Case: Idempotency & Duplicate Request Handling

The everyday problem: A user clicks "Pay Now", the request is slow, they click again — and they are charged twice. A mobile app sends an order, the network drops before the response arrives, the app retries — and two identical orders are created. A message queue redelivers a message after a worker crashed mid-processing — and the same email is sent twice. Networks are unreliable, users are impatient, and every retry mechanism (browsers, mobile apps, load balancers, message queues, Polly policies) can deliver the same request more than once. The challenge is making operations idempotent: performing the same request multiple times produces the same result as performing it once, with no duplicate side-effects.

Five approaches, from client-supplied keys to natural database constraints. The right one depends on whether you control the client, whether the operation has a natural unique identity, and whether you need to return the original response on a duplicate.

---

Way 1: Idempotency Keys (client-supplied unique token)

Best used for: Payment APIs, order creation, and any state-changing POST where the client can generate a unique key per logical operation — the industry-standard approach (Stripe, PayPal, and most payment providers require it).

The concept: The client generates a unique key (a GUID) for each logical operation and sends it in a header. The server records the key before processing. If the same key arrives again, the server recognises the duplicate and returns the original response instead of processing again. The key represents "this specific user intent", so retries of the same intent are collapsed.

// Client sends a stable key per logical operation (same key on retries)
// POST /api/payments
// Idempotency-Key: 9f86d081-... (generated once, reused on every retry of THIS payment)

public class IdempotencyMiddleware(RequestDelegate next, IDistributedCache cache)
{
    public async Task InvokeAsync(HttpContext context)
    {
        // Only guard unsafe methods
        if (context.Request.Method != HttpMethods.Post)
        { await next(context); return; }

        var key = context.Request.Headers["Idempotency-Key"].FirstOrDefault();
        if (string.IsNullOrEmpty(key))
        {
            context.Response.StatusCode = StatusCodes.Status400BadRequest;
            await context.Response.WriteAsync("Idempotency-Key header is required.");
            return;
        }

        var cacheKey = $"idem:{key}";
        var existing = await cache.GetStringAsync(cacheKey);
        if (existing is not null)
        {
            // Duplicate — replay the stored original response, do NOT process again
            var stored = JsonSerializer.Deserialize<StoredResponse>(existing)!;
            context.Response.StatusCode = stored.StatusCode;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(stored.Body);
            return;
        }

        // Capture the response so we can store and replay it
        var originalBody = context.Response.Body;
        using var buffer = new MemoryStream();
        context.Response.Body = buffer;

        await next(context); // process the request once

        buffer.Position = 0;
        var bodyText = await new StreamReader(buffer).ReadToEndAsync();
        buffer.Position = 0;
        await buffer.CopyToAsync(originalBody);
        context.Response.Body = originalBody;

        // Store the result for 24h so retries within that window are replayed
        if (context.Response.StatusCode is >= 200 and < 300)
        {
            var toStore = new StoredResponse(context.Response.StatusCode, bodyText);
            await cache.SetStringAsync(cacheKey, JsonSerializer.Serialize(toStore),
                new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(24) });
        }
    }
}
public record StoredResponse(int StatusCode, string Body);

Why it works: The key uniquely identifies the user's intent, independent of how many times the network delivers it. The first request processes and stores its result; every subsequent request with the same key short-circuits to the stored response. The client gets identical results whether it sent the request once or ten times.

Benchmark note: The overhead is one cache lookup per request (~sub-millisecond with Redis) — negligible against the protection it provides. The critical correctness detail is the race condition: two identical requests can arrive simultaneously before either has stored a result. The naive "check then process" leaves a gap. Production implementations store the key with a "processing" state atomically before processing (using Redis SET key value NX — set-if-not-exists) so the second concurrent request sees "in progress" and either waits or returns 409. Set a sensible expiry (Stripe uses 24h); keys cannot live forever.

---

Way 2: Database Dedup Table with Unique Constraint

Best used for: When you want the deduplication guarantee enforced by the database itself rather than a cache — strong consistency, survives cache loss, and the dedup record participates in the same transaction as the work.

The concept: A dedicated table has a unique constraint on the idempotency key. You insert the key in the same transaction as the actual work. If the key already exists, the unique-constraint violation tells you it is a duplicate — atomically, with the database as the arbiter. No separate "check then act" race.

-- The dedup table — unique constraint does the enforcement
CREATE TABLE processed_requests (
    idempotency_key UUID PRIMARY KEY,
    response_body   JSONB,
    status_code     INT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

public async Task<PaymentResult> ProcessPaymentAsync(Guid idempotencyKey, PaymentRequest req, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);
    try
    {
        // Attempt to claim the key — the unique constraint is the gatekeeper
        _db.ProcessedRequests.Add(new ProcessedRequest
        {
            IdempotencyKey = idempotencyKey,
            CreatedAt = DateTime.UtcNow
        });
        await _db.SaveChangesAsync(ct); // throws DbUpdateException if key already exists

        // First time — do the real work in the SAME transaction
        var result = await _paymentGateway.ChargeAsync(req, ct);

        // Persist the response so duplicates can replay it
        var record = await _db.ProcessedRequests.FindAsync([idempotencyKey], ct);
        record!.ResponseBody = JsonSerializer.Serialize(result);
        record.StatusCode    = 200;
        await _db.SaveChangesAsync(ct);

        await tx.CommitAsync(ct);
        return result;
    }
    catch (DbUpdateException) // unique-constraint violation = duplicate
    {
        await tx.RollbackAsync(ct);
        // Return the stored original response
        var existing = await _db.ProcessedRequests.AsNoTracking()
            .FirstAsync(r => r.IdempotencyKey == idempotencyKey, ct);
        return JsonSerializer.Deserialize<PaymentResult>(existing.ResponseBody!)!;
    }
}

Why it works: The unique constraint is enforced atomically by the database — there is no window between "check if exists" and "insert" because they are the same operation. Putting the key insert in the same transaction as the work means either both commit or both roll back: you never have a recorded key with no work done, or work done with no key recorded.

Benchmark note: Slightly heavier than a cache lookup (a real insert and the transaction), but rock-solid — it survives cache flushes and process crashes because the source of truth is the durable database. The transaction scope is the key design point: if the charge calls an external gateway, that call is not transactional, so you need the gateway itself to be idempotent too (most accept their own idempotency key — pass yours through). Periodically purge old rows (a nightly DELETE WHERE created_at < now() - interval '30 days') so the table does not grow unbounded.

---

Way 3: Natural Idempotency via Unique Business Keys

Best used for: Operations that already have a natural unique identity in the domain — one invoice per order, one registration per email, one booking per (seat, showtime) — where you do not need a separate idempotency token at all.

The concept: Instead of bolting on idempotency keys, you put a unique constraint on the natural business key. The operation becomes inherently idempotent because the domain only allows one. A second attempt to create the same logical entity hits the constraint and you handle it as "already exists".

-- The business rule IS the idempotency guarantee
CREATE UNIQUE INDEX ux_invoice_per_order ON invoices (order_id);
CREATE UNIQUE INDEX ux_user_email        ON users (lower(email));
CREATE UNIQUE INDEX ux_booking_seat_show ON bookings (showtime_id, seat_number);

public async Task<Invoice> CreateInvoiceAsync(int orderId, CancellationToken ct)
{
    // Idempotent because order_id is unique — two calls cannot both create an invoice
    var existing = await _db.Invoices.FirstOrDefaultAsync(i => i.OrderId == orderId, ct);
    if (existing is not null) return existing; // already created — return it

    try
    {
        var invoice = new Invoice { OrderId = orderId, CreatedAt = DateTime.UtcNow };
        _db.Invoices.Add(invoice);
        await _db.SaveChangesAsync(ct);
        return invoice;
    }
    catch (DbUpdateException) // lost the race to a concurrent request — fetch the winner's row
    {
        return await _db.Invoices.FirstAsync(i => i.OrderId == orderId, ct);
    }
}

Why it works: The domain constraint already says "only one of these can exist". By enforcing it at the database level and handling the violation gracefully, the operation is idempotent by construction — no extra keys, headers, or tables. The "check, then catch the constraint on the race" pattern handles both the common case and concurrent duplicates.

Benchmark note: This is the cleanest approach when a natural key exists — zero extra infrastructure, and the guarantee is intrinsic to the data model rather than an add-on that could be bypassed. The pre-check SELECT avoids most exception overhead (exceptions are expensive), while the catch handles the genuine concurrent-duplicate race. The limitation is obvious: it only applies when the operation has a natural unique identity. "Charge $50 to this card" has no natural key — the same charge legitimately can happen twice — so payments need explicit idempotency keys (Way 1/2). Use natural keys wherever the domain provides them; fall back to explicit keys where it does not.

---

Way 4: Conditional Writes / Optimistic Concurrency (ETags)

Best used for: Update operations where "apply this change only if the resource is still in the state I last saw" prevents both lost updates and duplicate-application of the same change — REST PUT/PATCH with ETags, inventory decrements, status transitions.

The concept: The client includes a version token (ETag / rowversion) representing the state it is updating. The write succeeds only if the current version matches. A duplicate retry carries the old version, which no longer matches after the first succeeded, so the duplicate is rejected — the change cannot be applied twice. This is the concurrency-token pattern applied to idempotency.

// HTTP conditional update — If-Match carries the version the client last saw
[HttpPut("products/{id}")]
public async Task<IActionResult> Update(int id, ProductUpdate update, CancellationToken ct)
{
    var ifMatch = Request.Headers.IfMatch.FirstOrDefault()?.Trim('"');
    if (ifMatch is null) return BadRequest("If-Match header required.");

    var product = await _db.Products.FindAsync([id], ct);
    if (product is null) return NotFound();

    // Tell EF the version the client based its update on
    _db.Entry(product).Property(p => p.RowVersion).OriginalValue = Convert.FromBase64String(ifMatch);

    product.Name  = update.Name;
    product.Price = update.Price;

    try
    {
        await _db.SaveChangesAsync(ct);
        // New ETag for the client's next update
        Response.Headers.ETag = $"\"{Convert.ToBase64String(product.RowVersion)}\"";
        return NoContent();
    }
    catch (DbUpdateConcurrencyException)
    {
        // The row changed since the client read it — could be a concurrent edit,
        // OR a duplicate of an update that already applied. Either way: reject.
        return StatusCode(StatusCodes.Status412PreconditionFailed,
            "The resource was modified. Re-read and retry.");
    }
}

Why it works: The version token ties the update to a specific prior state. Once the first update succeeds, the version changes, so a replayed duplicate (carrying the now-stale version) fails the condition and is rejected. The same change physically cannot be applied twice because the second attempt no longer matches the precondition.

Benchmark note: Near-zero overhead — the version check is part of the UPDATE ... WHERE RowVersion = @v the database already runs. This is the natural fit for resource updates (as opposed to creates) and doubles as protection against the lost-update problem. The trade-off is that it shifts work to the client: the client must read the current version, include it, and handle 412 by re-reading and retrying. It does not help with operations that are not "update an existing resource" — a fresh POST create has no prior version to condition on, so use idempotency keys there.

---

Way 5: Idempotent Consumers for Message Queues (dedup on message ID)

Best used for: Message-queue and event-driven systems where at-least-once delivery means consumers will occasionally receive the same message twice (after a crash, a redelivery, or a visibility-timeout expiry) and must not double-process it.

The concept: Every message carries a unique ID. The consumer records processed message IDs (in the same transaction as the side-effect) and skips any message ID it has already seen. This makes the consumer idempotent regardless of how many times the broker delivers the message — the foundation of reliable event-driven processing.

-- Track which messages have been processed
CREATE TABLE processed_messages (
    message_id  UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

public async Task HandleMessageAsync(QueueMessage message, CancellationToken ct)
{
    var messageId = Guid.Parse(message.Id);

    await using var tx = await _db.Database.BeginTransactionAsync(ct);
    try
    {
        // Claim the message ID — same transaction as the side-effect
        _db.ProcessedMessages.Add(new ProcessedMessage { MessageId = messageId });
        await _db.SaveChangesAsync(ct); // unique-PK violation if already processed

        // First time seeing this message — do the actual work
        var evt = JsonSerializer.Deserialize<OrderPlacedEvent>(message.Body)!;
        await _emailService.SendConfirmationAsync(evt.CustomerEmail, evt.OrderId, ct);
        await _inventory.ReserveAsync(evt.OrderId, ct);

        await tx.CommitAsync(ct);
        await message.AckAsync(ct); // tell the broker it is done
    }
    catch (DbUpdateException) // already processed — this is a redelivery
    {
        await tx.RollbackAsync(ct);
        await message.AckAsync(ct); // ack so the broker stops redelivering; do NOT reprocess
        _logger.LogInformation("Skipped duplicate message {MessageId}", messageId);
    }
}

Why it works: Recording the message ID in the same transaction as the side-effect means the side-effect and the "I processed this" record commit together atomically. A redelivered message hits the unique constraint and is skipped. Crucially, the consumer acknowledges the duplicate (so the broker stops redelivering) but does not repeat the work.

Benchmark note: This is what makes at-least-once delivery (the only delivery guarantee most queues can actually provide) safe to build on. The cost is one insert per message and a growing dedup table to prune. The subtle failure it does not fully solve: if the side-effect is a call to an external system (sending an email) and the process crashes after the email is sent but before the transaction commits, the email was sent but the message ID was not recorded — so on redelivery it sends again. Truly external side-effects need their own idempotency (e.g. the email provider deduping on a key you supply), or the transactional-outbox pattern to make the publish itself reliable. Within your own database, the transactional dedup is exactly-once in effect.

---

Choosing the right approach

• Payment / order POST, client can send a key → Idempotency keys (Way 1)
• Need DB-enforced guarantee, survives cache loss → Dedup table + unique constraint (Way 2)
• Operation has a natural unique identity → Natural business key (Way 3) — cleanest when available
• Updating an existing resource → Conditional write / ETag (Way 4)
• Consuming from a message queue → Idempotent consumer + message-ID dedup (Way 5)

The mental model: idempotency is about giving each logical operation a stable identity and refusing to perform the same identity twice. Where the domain supplies that identity naturally (one invoice per order), use it (Way 3). Where it does not, the client supplies it as an idempotency key (Way 1/2). For updates, the resource's version is the identity of the change (Way 4). For queues, the message ID is the identity (Way 5). The two recurring rules: record the identity atomically in the same transaction as the side-effect so you never get one without the other, and remember that any side-effect crossing a system boundary (payment gateway, email provider, another service) needs that downstream to be idempotent too — your own dedup only protects your own database.

Note: this is an informational engineering overview. Always validate idempotency behaviour under concurrent load and failure injection against your own systems.