Inter-Service Communication

Use Case: Inter-Service Communication

The everyday problem: Your monolith has grown into services — an Orders service, a Payments service, an Inventory service, a Notifications service. Now they need to talk to each other. The Orders service needs a customer's credit status from Payments; placing an order must reserve stock in Inventory and send an email via Notifications. Do them wrong and you get a distributed monolith: services tightly coupled, one slow service cascading failures everywhere, an outage in Notifications blocking order placement entirely. The challenge is choosing how services communicate — synchronously or asynchronously, point-to-point or via a broker — so the system stays responsive, resilient, and loosely coupled.

Five approaches spanning the synchronous/asynchronous spectrum. The single most important decision is per-interaction: does the caller need an answer right now (synchronous) or can the work happen eventually (asynchronous)? Getting this wrong is the root of most distributed-systems pain.

---

Way 1: REST over HTTP (HttpClientFactory)

Best used for: Synchronous request/response where the caller needs an immediate answer — fetching data, querying status, simple commands — between services that already speak HTTP. The default, lowest-friction choice.

The concept: One service calls another's HTTP API and waits for the response. The critical detail in .NET is using IHttpClientFactory — not new HttpClient() — to get pooled, correctly-managed connections with built-in resilience, and a typed client to keep call sites clean.

// Typed client registration with resilience (Polly via Microsoft.Extensions.Http.Resilience)
builder.Services.AddHttpClient<IPaymentClient, PaymentClient>(c =>
{
    c.BaseAddress = new Uri("https://payments-service/");   // service discovery name
    c.Timeout = TimeSpan.FromSeconds(10);
})
.AddStandardResilienceHandler(); // retry + circuit breaker + timeout out of the box (.NET 8+)

// The typed client
public class PaymentClient(HttpClient http) : IPaymentClient
{
    public async Task<CreditStatus> GetCreditStatusAsync(int customerId, CancellationToken ct)
    {
        var resp = await http.GetAsync($"api/customers/{customerId}/credit", ct);
        resp.EnsureSuccessStatusCode();
        return (await resp.Content.ReadFromJsonAsync<CreditStatus>(cancellationToken: ct))!;
    }
}

// Usage in the Orders service
var credit = await _paymentClient.GetCreditStatusAsync(order.CustomerId, ct);
if (!credit.IsApproved) return BadRequest("Credit not approved.");

Why it works: HTTP/JSON is universal, debuggable (curl, Postman, browser dev tools), and every language speaks it. IHttpClientFactory solves the notorious socket-exhaustion problem of raw HttpClient by pooling and rotating handlers, and the standard resilience handler adds retry/circuit-breaker/timeout without custom code.

Benchmark note: JSON serialization and HTTP framing make REST the heaviest-per-call of the synchronous options — typically single-digit to low-tens of milliseconds per call on a fast network, dominated by serialization and connection overhead. Fine for most interactions; noticeable when one request fans out to many downstream calls. The deeper risk is synchronous coupling: if Orders calls Payments calls Fraud calls..., one slow link in the chain slows everything, and a downstream outage propagates upward. Mitigate with timeouts and circuit breakers (the resilience use case), and question whether each call really needs to be synchronous — many do not.

---

Way 2: gRPC (high-performance binary RPC)

Best used for: High-throughput, low-latency synchronous calls between internal services — chatty service-to-service traffic, streaming, polyglot back-ends — where REST's JSON overhead is a measurable cost.

The concept: gRPC uses HTTP/2 and Protocol Buffers (a compact binary format) with a contract-first .proto file that generates strongly-typed clients and servers. It supports unary calls and bidirectional streaming, and the binary encoding plus HTTP/2 multiplexing make it substantially faster and lighter than REST/JSON.

// payments.proto — the contract, shared between services
// service PaymentService {
//   rpc GetCreditStatus (CreditRequest) returns (CreditResponse);
//   rpc StreamTransactions (CustomerRef) returns (stream Transaction); // server streaming
// }

// Server implementation (Payments service)
public class PaymentGrpcService : PaymentService.PaymentServiceBase
{
    public override async Task<CreditResponse> GetCreditStatus(
        CreditRequest request, ServerCallContext context)
    {
        var status = await _credit.CheckAsync(request.CustomerId);
        return new CreditResponse { IsApproved = status.Approved, Limit = (double)status.Limit };
    }
}

// Client (Orders service) — generated strongly-typed client
builder.Services.AddGrpcClient<PaymentService.PaymentServiceClient>(o =>
    o.Address = new Uri("https://payments-service"))
    .AddStandardResilienceHandler();

var response = await _paymentGrpc.GetCreditStatusAsync(
    new CreditRequest { CustomerId = order.CustomerId }, cancellationToken: ct);

Why it works: Protocol Buffers serialize to a fraction of the size of JSON and parse far faster; HTTP/2 multiplexes many calls over one connection and avoids head-of-line blocking. The .proto contract gives compile-time type safety on both ends and generates the boilerplate, so drift between client and server is caught at build time.

Benchmark note: gRPC is commonly 5–10× faster than REST/JSON for the same payload and uses far less bandwidth — the gap widens with payload size and call frequency. It shines for internal, high-volume, chatty paths and for streaming (which REST handles awkwardly). The trade-offs: it is binary (not human-readable without tooling), browser support requires gRPC-Web with a proxy, and the .proto contract adds a build step. Use gRPC for internal service-to-service hot paths; keep REST at the public edge where universality and debuggability matter more than raw speed.

---

Way 3: Asynchronous Messaging (queue — point-to-point commands)

Best used for: Fire-and-handle-later commands where the caller does not need an immediate result — "process this order", "send this email", "resize this image" — and you want the caller decoupled from the handler's availability and speed.

The concept: Instead of calling another service directly, the producer puts a message on a queue (Azure Service Bus, RabbitMQ, AWS SQS) and returns immediately. A consumer service pulls the message and processes it on its own schedule. The queue buffers between them, so the producer is unaffected by the consumer being slow, busy, or temporarily down.

// Producer (Orders) — enqueue a command and move on; does NOT wait for processing
public async Task PlaceOrderAsync(Order order, CancellationToken ct)
{
    await _db.SaveOrderAsync(order, ct);

    // Hand off downstream work asynchronously — Orders does not block on it
    await _bus.SendAsync(new ProcessPaymentCommand(order.Id, order.Total), ct);
    await _bus.SendAsync(new ReserveInventoryCommand(order.Id, order.Items), ct);
    await _bus.SendAsync(new SendConfirmationCommand(order.Id, order.CustomerEmail), ct);
    // Order placement returns immediately; the rest happens when consumers pick it up
}

// Consumer (Payments) — processes at its own pace
public class PaymentCommandConsumer(IServiceProvider services) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var msg in _bus.ReceiveAsync<ProcessPaymentCommand>(ct))
        {
            using var scope = services.CreateScope();
            try
            {
                await scope.ServiceProvider.GetRequiredService<IPaymentProcessor>()
                    .ChargeAsync(msg.OrderId, msg.Amount, ct);
                await msg.AckAsync(ct);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Payment failed; message will retry / dead-letter");
                await msg.NackAsync(ct); // redelivery, then dead-letter after N attempts
            }
        }
    }
}

Why it works: The queue decouples producer and consumer in time and availability. If Payments is down for 5 minutes, orders still get placed — the commands wait in the queue and process when Payments recovers. The producer's latency and reliability no longer depend on the consumer's. This is the antidote to synchronous-coupling cascades.

Benchmark note: Asynchronous messaging trades immediate consistency for resilience and scalability — the order is placed before payment is processed, so the system is eventually consistent and the UI must reflect "payment pending". Queues absorb traffic spikes (the buffer flattens bursts) and let you scale consumers independently. The cost is operational complexity and the at-least-once delivery guarantee, which means consumers must be idempotent (see the idempotency use case). Use this for any downstream work that does not need a synchronous answer — which, on reflection, is most of it.

---

Way 4: Event-Driven / Publish-Subscribe (broadcast facts)

Best used for: One service needs to announce that something happened and any number of other services may want to react — "OrderPlaced", "PaymentCompleted", "UserRegistered" — without the publisher knowing or caring who is listening.

The concept: The difference from Way 3 is command vs event. A command ("ProcessPayment") targets one handler and says "do this". An event ("OrderPlaced") is a broadcast statement of fact published to a topic; zero or more subscribers react however they wish. The publisher is fully decoupled — it does not know its subscribers exist. New subscribers can be added later with no change to the publisher.

// Publisher (Orders) — announces a fact; knows nothing about who reacts
public async Task PlaceOrderAsync(Order order, CancellationToken ct)
{
    await _db.SaveOrderAsync(order, ct);
    await _bus.PublishAsync(new OrderPlacedEvent(order.Id, order.CustomerId, order.Total), ct);
    // Done. Whoever cares about OrderPlaced will react on their own.
}

// Independent subscribers — each reacts to the SAME event for its own reasons
public class InventorySubscriber  // reserves stock
{
    public Task Handle(OrderPlacedEvent e, CancellationToken ct)
        => _inventory.ReserveAsync(e.OrderId, ct);
}
public class NotificationSubscriber  // sends a confirmation email
{
    public Task Handle(OrderPlacedEvent e, CancellationToken ct)
        => _email.SendOrderConfirmationAsync(e.CustomerId, e.OrderId, ct);
}
public class AnalyticsSubscriber  // updates a sales dashboard — added LATER, no Orders change
{
    public Task Handle(OrderPlacedEvent e, CancellationToken ct)
        => _analytics.RecordSaleAsync(e.OrderId, e.Total, ct);
}
// Each subscriber has its own queue subscription to the OrderPlaced topic,
// so a failure in one (e.g. email down) does not affect the others.

Why it works: Publish-subscribe inverts the dependency: instead of the publisher knowing all the things that must happen (and being coupled to each), it simply states what happened. Subscribers own their own reactions. This is the most loosely-coupled pattern — you can add, remove, or change reactions without ever touching the publisher, which is what makes event-driven architectures extensible.

Benchmark note: The scalability and decoupling are maximal, but so is the difficulty of reasoning about the system: there is no single place that shows "everything that happens when an order is placed" — the logic is distributed across subscribers. Debugging requires distributed tracing (correlation IDs flowing through events) to follow a single business operation across services. Each subscriber needs its own dead-letter handling and idempotency. Events should carry enough data for subscribers to act without calling back (avoid the "event then N synchronous callbacks" anti-pattern). Use pub/sub when multiple independent reactions to a fact are the norm; use commands (Way 3) when there is exactly one intended handler.

---

Way 5: Outbox Pattern (reliable publish — no lost or phantom events)

Best used for: Guaranteeing that an event is published if and only if the database change that triggered it committed — eliminating the dual-write problem that silently corrupts every naive event-driven system.

The concept: The dual-write problem: if you save the order to the database and then publish "OrderPlaced" to the broker as two separate operations, a crash between them either loses the event (order saved, no event — subscribers never react) or creates a phantom (event published, save rolled back — subscribers react to an order that does not exist). The outbox pattern fixes this by writing the event into an outbox table in the same database transaction as the business change. A separate process then reads the outbox and publishes to the broker, marking events as sent. The event and the data commit atomically.

-- The outbox table lives in the same database as the business data
CREATE TABLE outbox_messages (
    id           UUID PRIMARY KEY,
    type         TEXT NOT NULL,
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at TIMESTAMPTZ NULL
);

// 1. Producer writes business data AND the event in ONE transaction — atomic
public async Task PlaceOrderAsync(Order order, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);

    _db.Orders.Add(order);
    _db.OutboxMessages.Add(new OutboxMessage
    {
        Id = Guid.NewGuid(),
        Type = nameof(OrderPlacedEvent),
        Payload = JsonSerializer.Serialize(new OrderPlacedEvent(order.Id, order.CustomerId, order.Total))
    });

    await _db.SaveChangesAsync(ct); // order + outbox row commit together — or neither does
    await tx.CommitAsync(ct);
}

// 2. A separate publisher polls the outbox and relays to the broker
public class OutboxPublisher(IServiceProvider services, IMessageBus bus) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            using var scope = services.CreateScope();
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

            var pending = await db.OutboxMessages
                .Where(m => m.ProcessedAt == null)
                .OrderBy(m => m.CreatedAt)
                .Take(100)
                .ToListAsync(ct);

            foreach (var msg in pending)
            {
                await bus.PublishRawAsync(msg.Type, msg.Payload, ct); // relay to broker
                msg.ProcessedAt = DateTime.UtcNow;                    // mark sent
            }
            await db.SaveChangesAsync(ct);
            await Task.Delay(TimeSpan.FromSeconds(1), ct);
        }
    }
}

Why it works: Because the event is written in the same transaction as the data, there is no window where one exists without the other — they commit atomically or roll back together. The publisher relaying from the outbox guarantees at-least-once delivery to the broker (a crash mid-publish just leaves the row unprocessed for the next poll), and consumer-side idempotency handles the resulting possible duplicates. This is the correct foundation for every reliable event-driven system.

Benchmark note: The outbox adds a small write (one extra row per event) and a polling publisher with a slight delivery latency (sub-second to a few seconds depending on poll interval) — a minor cost for eliminating the entire class of dual-write data-loss bugs. To reduce latency and database polling load at scale, change-data-capture (Debezium, or SQL Server CDC) can tail the transaction log and publish outbox rows in near-real-time instead of polling. Libraries like MassTransit and NServiceBus implement the outbox for you. The rule this enforces across distributed systems: never dual-write to a database and a broker as separate operations — always go through the outbox.

---

Choosing the right approach

• Need an immediate answer, standard interaction → REST / HttpClientFactory (Way 1)
• Need an immediate answer, high-volume internal hot path → gRPC (Way 2)
• Fire-and-handle-later command, one handler → Async queue (Way 3)
• Announce a fact, many independent reactions → Event-driven pub/sub (Way 4)
• Reliable event publishing (no lost/phantom events) → Outbox pattern (Way 5) — wraps Way 3/4

The decision spine: for every interaction, first ask synchronous or asynchronous? — does the caller genuinely need the answer before it can continue (synchronous: Way 1/2), or can the work happen eventually (asynchronous: Way 3/4)? Default to asynchronous wherever the business allows, because synchronous chains couple availability and cascade failures. Among synchronous options, REST at the edge and for simplicity, gRPC for internal high-throughput. Among asynchronous, commands (Way 3) when there is one intended handler, events (Way 4) when many services react to a fact. And whenever an event must reliably follow a database change, route it through the outbox (Way 5) — the dual-write problem is silent, common, and only the outbox actually solves it.

Note: this is an informational engineering overview. Performance figures are illustrative and depend on payload size, network, serialization, and infrastructure. Always measure against your own environment.