Distributed Transactions & Data Consistency

Use Case: Distributed Transactions & Data Consistency Across Services

The everyday problem: Placing an order must charge the customer (Payments service), reserve stock (Inventory service), and create a shipment (Shipping service). In a monolith with one database, a single transaction makes this atomic — all succeed or all roll back. But once these are separate services with separate databases, there is no shared transaction. What happens when payment succeeds, inventory reservation succeeds, but shipment creation fails? The customer is charged for an order that will never ship. You cannot use a traditional BEGIN TRANSACTION ... COMMIT across service boundaries. The challenge is maintaining data consistency across multiple services and databases when a single atomic transaction is impossible.

Five approaches to consistency without distributed transactions. The central insight: in distributed systems you usually trade immediate consistency for eventual consistency, managed explicitly through compensation and reliable messaging — because the classic alternative (two-phase commit) does not scale and is rarely usable across modern services.

---

Way 1: SAGA — Orchestration (a central coordinator)

Best used for: Multi-step business processes spanning several services where you want one place that defines and controls the whole flow — order fulfilment, booking systems, onboarding pipelines — with clear visibility into where each transaction is.

The concept: A SAGA breaks a distributed transaction into a sequence of local transactions, one per service. In the orchestration variant, a central coordinator (the orchestrator) tells each service what to do, in order, and waits for each to confirm. If any step fails, the orchestrator runs compensating transactions — explicit "undo" operations — for the steps that already succeeded, in reverse order. There is no rollback; there is deliberate, business-defined compensation.

// Orchestrator drives the saga step by step and compensates on failure
public class OrderSaga(IPaymentClient payments, IInventoryClient inventory, IShippingClient shipping,
    ILogger<OrderSaga> logger)
{
    public async Task<SagaResult> ExecuteAsync(Order order, CancellationToken ct)
    {
        var completed = new Stack<Func<Task>>(); // compensations to run if a later step fails

        try
        {
            // Step 1: charge payment
            var payment = await payments.ChargeAsync(order.CustomerId, order.Total, ct);
            completed.Push(() => payments.RefundAsync(payment.Id, ct)); // its compensation

            // Step 2: reserve inventory
            var reservation = await inventory.ReserveAsync(order.Items, ct);
            completed.Push(() => inventory.ReleaseAsync(reservation.Id, ct));

            // Step 3: create shipment
            var shipment = await shipping.CreateAsync(order, ct);
            completed.Push(() => shipping.CancelAsync(shipment.Id, ct));

            return SagaResult.Success(order.Id);
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Saga failed for order {OrderId}; compensating", order.Id);

            // Run compensations in REVERSE order for the steps that succeeded
            while (completed.Count > 0)
            {
                try { await completed.Pop()(); }
                catch (Exception cex)
                {
                    // Compensation itself failed — critical: needs alerting / manual intervention
                    logger.LogCritical(cex, "Compensation failed for order {OrderId}", order.Id);
                }
            }
            return SagaResult.Failed(order.Id, ex.Message);
        }
    }
}

Why it works: Each service commits its own local transaction (which it can do atomically), and the orchestrator coordinates the sequence and the undo. Because there is one coordinator, the entire flow is visible and centrally managed — you can see exactly which step a given order reached, and the compensation logic is explicit and ordered.

Benchmark note: Orchestration's strength is clarity: one place defines the workflow, making complex flows understandable and the state queryable. Its weakness is that the orchestrator is a central dependency and a potential single point of failure, so for serious use it should itself be durable — typically implemented on a workflow engine (Azure Durable Functions, Temporal, or a saga library like MassTransit's state machine) so the saga's own state survives crashes and resumes mid-flow. The hardest part is not the happy path but compensation design: refunding a payment is clean, but some actions have no perfect undo (an email was sent, a physical process started), so compensations are often "best-effort plus alert". Compensations must also be idempotent, since they may be retried.

---

Way 2: SAGA — Choreography (events, no central coordinator)

Best used for: Distributed transactions where you want maximum decoupling and no central coordinator — each service knows only its own part and reacts to events, suiting event-driven architectures and autonomous teams.

The concept: Instead of a central orchestrator, each service listens for events, does its local transaction, and publishes an event announcing the result — which the next service reacts to. The flow emerges from the chain of events rather than being directed from one place. Failures trigger compensation events that ripple back.

// Each service reacts to events and emits the next — no central controller

// Payments service: reacts to OrderCreated
public class PaymentHandler
{
    public async Task On(OrderCreatedEvent e, CancellationToken ct)
    {
        try
        {
            await _payments.ChargeAsync(e.CustomerId, e.Total, ct);
            await _bus.PublishAsync(new PaymentCompletedEvent(e.OrderId), ct); // → triggers inventory
        }
        catch
        {
            await _bus.PublishAsync(new PaymentFailedEvent(e.OrderId), ct);    // → triggers order cancel
        }
    }
}

// Inventory service: reacts to PaymentCompleted
public class InventoryHandler
{
    public async Task On(PaymentCompletedEvent e, CancellationToken ct)
    {
        try
        {
            await _inventory.ReserveAsync(e.OrderId, ct);
            await _bus.PublishAsync(new InventoryReservedEvent(e.OrderId), ct); // → triggers shipping
        }
        catch
        {
            // Compensation ripples BACKWARD: tell Payments to refund
            await _bus.PublishAsync(new InventoryFailedEvent(e.OrderId), ct);   // → triggers refund
        }
    }
}

// Payments also listens for compensation events
public class PaymentCompensationHandler
{
    public Task On(InventoryFailedEvent e, CancellationToken ct)
        => _payments.RefundAsync(e.OrderId, ct); // undo the earlier charge
}

Why it works: No central coordinator means no single point of control or failure — each service is autonomous, owning only its local transaction and its reactions. Services are maximally decoupled: adding a new participant means subscribing to the relevant event, with no change to existing services. It fits naturally on top of an event bus.

Benchmark note: Choreography maximises decoupling and autonomy but sacrifices visibility: there is no single place that shows the whole workflow — the logic is distributed across event handlers in different services, which makes the overall flow hard to understand and debug. Tracing a single order's journey requires correlation IDs flowing through every event and distributed tracing to reconstruct the path. It also risks cyclic complexity as event chains grow ("who publishes what, and what listens?" becomes a tangled graph). Rule of thumb: choreography for simple sagas (2–3 steps) where decoupling is prized; orchestration (Way 1) for complex sagas (4+ steps) where understanding and controlling the flow matters more. Both require reliable event delivery — which means the outbox pattern (Way 4).

---

Way 3: Eventual Consistency with Reconciliation

Best used for: Systems that can tolerate temporary inconsistency and need a safety net to detect and fix discrepancies that slip through — financial reconciliation, inventory counts, any system where "eventually correct" is acceptable and a periodic check catches what compensation missed.

The concept: Accept that the system will be temporarily inconsistent (one service updated, another not yet) and design for it explicitly: the UI shows "pending" states, and a periodic reconciliation process compares the services' data, detects discrepancies, and corrects them. This is the backstop that catches the cases where sagas and compensations failed silently.

// A reconciliation job that detects and repairs cross-service inconsistencies
public class OrderReconciliationJob(IOrderRepo orders, IPaymentRepo payments,
    IInventoryRepo inventory, ILogger<OrderReconciliationJob> logger)
{
    public async Task ReconcileAsync(CancellationToken ct)
    {
        // Find orders stuck mid-flow longer than expected
        var stuck = await orders.GetPendingOlderThanAsync(TimeSpan.FromMinutes(30), ct);

        foreach (var order in stuck)
        {
            var paid     = await payments.IsChargedAsync(order.Id, ct);
            var reserved = await inventory.IsReservedAsync(order.Id, ct);

            switch (paid, reserved)
            {
                case (true, true):
                    // Both done but order never marked complete — finish it
                    await orders.MarkCompleteAsync(order.Id, ct);
                    logger.LogInformation("Reconciled order {Id}: completed", order.Id);
                    break;

                case (true, false):
                    // Charged but not reserved and stuck — refund (compensate the orphan)
                    await payments.RefundAsync(order.Id, ct);
                    await orders.MarkFailedAsync(order.Id, ct);
                    logger.LogWarning("Reconciled order {Id}: refunded orphaned charge", order.Id);
                    break;

                case (false, true):
                    // Reserved but never charged — release the orphaned reservation
                    await inventory.ReleaseAsync(order.Id, ct);
                    await orders.MarkFailedAsync(order.Id, ct);
                    break;
            }
        }
    }
}

Why it works: Distributed systems fail in messy ways — a service crashes after committing but before publishing its event, a compensation silently fails, a message is lost. Reconciliation accepts this reality and provides a periodic sweep that detects the resulting inconsistencies (orphaned charges, dangling reservations, stuck orders) and resolves them. It is the safety net beneath the sagas.

Benchmark note: Reconciliation is not a primary consistency mechanism — it is the essential backstop that every serious distributed system needs in addition to sagas, because sagas and compensations themselves can fail. Financial systems depend on it heavily (end-of-day reconciliation is standard practice). The design work is defining what "inconsistent" means for each cross-service invariant and how to resolve each case safely (the resolution must be idempotent — reconciliation runs repeatedly). The trade-off is the consistency window: data can be wrong for up to one reconciliation interval, so the UI must communicate pending states honestly and the business must tolerate that window. Tune frequency to the cost of inconsistency: every few minutes for orders, continuously for high-value financial flows.

---

Way 4: Transactional Outbox (reliable events — the foundation)

Best used for: Guaranteeing that the events sagas depend on are never lost — making the local-transaction-plus-publish step atomic, which every saga (orchestration or choreography) silently relies on to work correctly.

The concept: Sagas depend on events reliably following state changes. But writing to your database and publishing to a message broker are two separate systems — if you do them as two operations, a crash between them loses the event (state changed, no event → the saga stalls) or creates a phantom (event sent, state rolled back → the saga acts on nothing). The outbox writes the event into an outbox table in the same local transaction as the state change; a separate relay publishes from the outbox. Event and state commit atomically.

// The local transaction writes business state AND the outgoing event atomically
public async Task ReserveInventoryAsync(int orderId, List<Item> items, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);

    // 1. The local state change
    _db.Reservations.Add(new Reservation { OrderId = orderId, Items = items });

    // 2. The event to publish — written to the SAME database in the SAME transaction
    _db.Outbox.Add(new OutboxMessage
    {
        Id = Guid.NewGuid(),
        Type = nameof(InventoryReservedEvent),
        Payload = JsonSerializer.Serialize(new InventoryReservedEvent(orderId))
    });

    await _db.SaveChangesAsync(ct); // both commit together — atomic
    await tx.CommitAsync(ct);
    // If the process crashes now, the event is safely in the outbox; the relay will publish it.
}

// A relay publishes outbox rows to the broker and marks them sent (at-least-once)
public class OutboxRelay(IServiceProvider services, IMessageBus bus) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            using var scope = services.CreateScope();
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

            var pending = await db.Outbox.Where(m => m.ProcessedAt == null)
                .OrderBy(m => m.CreatedAt).Take(100).ToListAsync(ct);

            foreach (var m in pending)
            {
                await bus.PublishRawAsync(m.Type, m.Payload, ct);
                m.ProcessedAt = DateTime.UtcNow;
            }
            await db.SaveChangesAsync(ct);
            await Task.Delay(TimeSpan.FromMilliseconds(500), ct);
        }
    }
}

Why it works: Because the event is written in the same local transaction as the state change, there is no gap where one exists without the other — atomicity is preserved within the single database. The relay then guarantees the event reaches the broker at least once (a crash mid-publish just leaves the row for the next pass), and consumer idempotency handles the resulting possible duplicates. This is what makes the events sagas rely on actually reliable.

Benchmark note: The outbox is the unglamorous foundation that makes everything above actually work — without it, sagas built on naive dual-writes lose events and stall in production in ways that are intermittent and agonising to diagnose. The cost is a small write per event and sub-second publish latency (polling interval); change-data-capture (Debezium / SQL Server CDC) tailing the transaction log reduces latency and database load at scale. Libraries like MassTransit and NServiceBus provide the outbox built-in. The absolute rule it enforces: never write to your database and publish to a broker as two separate operations — route every event through the outbox so it shares the transaction.

---

Way 5: Two-Phase Commit (and why to usually avoid it)

Best used for: The rare cases requiring genuine immediate cross-resource atomicity within a controlled environment — typically multiple databases or a database plus a transactional queue under one transaction coordinator, on infrastructure that supports it. Included largely to explain why the patterns above exist as alternatives.

The concept: Two-phase commit (2PC) uses a transaction coordinator across multiple resources. Phase 1 (prepare): the coordinator asks every participant "can you commit?" and each does the work tentatively and votes yes/no. Phase 2 (commit): if all voted yes, the coordinator tells everyone to commit; if any voted no, everyone rolls back. It provides true distributed atomicity — but at a steep cost.

// Conceptual 2PC via a coordinator (e.g. MSDTC with TransactionScope across two SQL DBs)
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
    // Both operations enlist in the SAME distributed transaction
    await _databaseA.UpdateAsync(...); // enlists resource A
    await _databaseB.UpdateAsync(...); // enlists resource B

    scope.Complete(); // triggers the two-phase commit across A and B
    // Coordinator: PREPARE both → if both vote yes, COMMIT both; else ROLLBACK both
}
// Works only where a distributed transaction coordinator exists and all resources support it.
// Does NOT work across HTTP microservices, most cloud-managed databases, or heterogeneous systems.

Why it (sometimes) works — and usually does not: 2PC delivers real immediate atomicity, which is why it is appealing on paper. But it requires a transaction coordinator that all participants support and enlist with, which is unavailable across independent microservices, most managed cloud databases, and heterogeneous tech stacks. Worse, it is a blocking protocol: during the prepare phase, participants hold locks and wait for the coordinator — if the coordinator crashes at the wrong moment, participants can be left locked indefinitely ("in-doubt" transactions). It also scales poorly, since every participant's latency and availability are coupled into one synchronous protocol.

Benchmark note: 2PC's blocking nature and coordinator dependency make it a poor fit for modern distributed systems — it couples availability (every participant must be up for any commit), holds locks across network round-trips (hurting throughput), and fails badly under coordinator failure. This is precisely why the industry moved to sagas and eventual consistency: they trade immediate consistency for availability and resilience, which is almost always the right trade in distributed systems. Reserve 2PC for the narrow case of a few homogeneous resources under one coordinator needing strict atomicity (e.g. two databases via MSDTC in an enterprise environment). For anything crossing service boundaries, use a saga (Ways 1–2) with the outbox (Way 4) and reconciliation (Way 3).

---

Choosing the right approach

• Complex multi-step flow, want central control + visibility → SAGA orchestration (Way 1)
• Simple flow, maximum decoupling, event-driven → SAGA choreography (Way 2)
• Safety net for discrepancies that slip through → Eventual consistency + reconciliation (Way 3)
• Reliable events underpinning any saga → Transactional outbox (Way 4) — the foundation
• Few homogeneous resources, true atomicity, controlled infra → 2PC (Way 5) — rarely

The mental model: once a transaction crosses service/database boundaries, the single atomic commit is gone and you cannot get it back cheaply. So you decompose the operation into per-service local transactions and manage consistency explicitly: a saga sequences them and compensates on failure (orchestration for control, choreography for decoupling); the outbox guarantees the events the saga depends on are never lost; reconciliation sweeps up the inconsistencies that still slip through; and you embrace eventual consistency, surfacing "pending" states honestly in the UI. Two-phase commit, the one approach offering immediate atomicity, is usually the wrong tool because its blocking, coordinator-bound nature does not fit independent services. The recurring themes: compensations replace rollbacks and must be idempotent; events must be reliable (outbox); and the system must be designed to be correct eventually, with a reconciliation backstop, rather than perfectly atomic at every instant.

Note: this is an informational engineering overview. Consistency, latency, and failure behaviour depend on infrastructure and design choices. Always validate saga flows and compensation logic with failure injection against your own systems.