Concurrency & Parallel Processing

Use Case: Concurrency & Parallel Processing

The everyday problem: You need to call 50 APIs, process 10,000 images, or import a million records — and doing them one at a time takes forever while the CPU and network sit mostly idle. Or a background pipeline must process a continuous stream of work with several stages. Or two requests update the same row and one silently overwrites the other. The challenge is doing many things at once to use available resources fully — without spawning unbounded work that exhausts memory and connections, without race conditions that corrupt shared state, and without the subtle bugs that make concurrent code so notoriously hard to get right.

Five approaches, each suited to a different shape of concurrent work: awaiting many independent tasks, bounded parallel iteration, multi-stage pipelines, producer/consumer streaming, and gating access to limited resources.

---

Way 1: Task.WhenAll (await many independent operations)

Best used for: A known, bounded set of independent asynchronous operations you want to run concurrently and wait for all of them — fetching from several services at once, loading multiple pieces of a page in parallel.

The concept: Start all the operations without awaiting each individually (so they run concurrently), collect the tasks, and await Task.WhenAll to wait for the whole set. Total time becomes roughly the slowest single operation rather than the sum of all of them.

// SEQUENTIAL — each awaited in turn; total = sum of all latencies
var profile  = await _users.GetProfileAsync(userId, ct);      // 100ms
var orders   = await _orders.GetRecentAsync(userId, ct);      // 150ms
var recs     = await _recommendations.GetAsync(userId, ct);   // 120ms
// Total ≈ 370ms (100 + 150 + 120)

// CONCURRENT — all started together; total = slowest one
var profileTask = _users.GetProfileAsync(userId, ct);
var ordersTask  = _orders.GetRecentAsync(userId, ct);
var recsTask    = _recommendations.GetAsync(userId, ct);

await Task.WhenAll(profileTask, ordersTask, recsTask);
var dashboard = new Dashboard(
    profileTask.Result, ordersTask.Result, recsTask.Result); // safe: all completed
// Total ≈ 150ms (the slowest), not 370ms

// Projecting a collection into concurrent tasks
var ids = new[] { 1, 2, 3, 4, 5 };
var results = await Task.WhenAll(ids.Select(id => _api.FetchAsync(id, ct)));

Why it works: Asynchronous I/O operations spend most of their time waiting (for the network, the database), not using the CPU. Starting them together lets all that waiting overlap — three 100ms calls that each spend 99ms waiting can almost entirely overlap, finishing in ~100ms total instead of 300ms.

Benchmark note: For I/O-bound fan-out, Task.WhenAll turns sum-of-latencies into max-of-latencies — often a 3–10× wall-clock improvement for a handful of calls. Two cautions. First, it does not limit concurrency: Task.WhenAll over 10,000 items starts all 10,000 at once, which can exhaust connections, sockets, or the downstream service — for large or unbounded sets, use Way 2 instead. Second, exception handling: if multiple tasks fail, await surfaces only the first exception; inspect the individual tasks (or the AggregateException on the returned task) to see all failures. Only use .Result after WhenAll has completed — never before, or it blocks.

---

Way 2: Parallel.ForEachAsync (bounded parallel iteration)

Best used for: Processing a large collection concurrently with a controlled degree of parallelism — calling an API for each of 10,000 items, but only 20 at a time so you neither overwhelm the downstream nor exhaust your own resources.

The concept: Parallel.ForEachAsync (.NET 6+) iterates a collection running an async body concurrently, but caps the number of simultaneous operations via MaxDegreeOfParallelism. It is the bounded version of "do this for every item at once" — concurrency with a safety limit built in.

// Process 10,000 items, but never more than 20 concurrently
await Parallel.ForEachAsync(
    items,
    new ParallelOptions
    {
        MaxDegreeOfParallelism = 20,   // the crucial limit — bounds resource use
        CancellationToken      = ct
    },
    async (item, token) =>
    {
        var enriched = await _api.EnrichAsync(item, token); // runs ≤20 at a time
        await _db.SaveEnrichedAsync(enriched, token);
    });
// 10,000 items / 20 concurrent — fast, but the downstream sees at most 20 in-flight

// Collecting results safely from parallel work — use a thread-safe collection
var results = new ConcurrentBag<Result>();
await Parallel.ForEachAsync(items, options, async (item, token) =>
{
    var r = await ProcessAsync(item, token);
    results.Add(r); // ConcurrentBag is safe for parallel writes; a plain List is NOT
});

Why it works: It captures the benefit of concurrency (many items processed simultaneously) while the degree-of-parallelism cap prevents the failure mode of unbounded fan-out — it self-throttles to your chosen limit, keeping connection pools, memory, and downstream load within safe bounds. The framework handles partitioning the work and managing the concurrent operations.

Benchmark note: The right MaxDegreeOfParallelism depends on the work. For I/O-bound work (API calls), it can be high (tens to low hundreds) since each operation mostly waits. For CPU-bound work, exceeding the core count is counterproductive — context-switching overhead with no gain. For work that hits a downstream with its own limits, match its capacity (no point sending 200 concurrent calls to a service that allows 20). The common bug here is writing to a non-thread-safe collection (List<T>, Dictionary) from the parallel body — that corrupts state or throws; always use ConcurrentBag/ConcurrentDictionary or other thread-safe collections for shared results.

---

Way 3: TPL Dataflow (multi-stage processing pipelines)

Best used for: Workflows with distinct stages where each stage has different concurrency needs — download (I/O-bound, high concurrency) → transform (CPU-bound, core-limited) → save (database, bounded) — and you want items to flow through the stages with independent throttling per stage.

The concept: TPL Dataflow (System.Threading.Tasks.Dataflow) models processing as a network of linked blocks. Each block performs one stage, has its own degree of parallelism and buffer, and passes results to the next block. Items flow through the pipeline; each stage processes at its own optimal rate, and back-pressure propagates automatically when a downstream stage is slower.

using System.Threading.Tasks.Dataflow;

// Stage 1: download — I/O-bound, allow high concurrency
var download = new TransformBlock<string, byte[]>(
    async url => await _http.GetByteArrayAsync(url),
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 50, BoundedCapacity = 100 });

// Stage 2: process — CPU-bound, limit to core count
var process = new TransformBlock<byte[], Image>(
    bytes => ResizeAndCompress(bytes),
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = Environment.ProcessorCount, BoundedCapacity = 100 });

// Stage 3: save — database, keep concurrency modest
var save = new ActionBlock<Image>(
    async img => await _storage.SaveAsync(img),
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 10, BoundedCapacity = 100 });

// Link the stages into a pipeline, propagating completion
var link = new DataflowLinkOptions { PropagateCompletion = true };
download.LinkTo(process, link);
process.LinkTo(save, link);

// Feed the pipeline — back-pressure blocks here if downstream buffers fill
foreach (var url in imageUrls)
    await download.SendAsync(url);
download.Complete();
await save.Completion; // wait for the whole pipeline to drain

Why it works: Each stage is tuned independently — the I/O-heavy download runs 50-wide while the CPU-bound transform runs only core-count-wide, so neither bottlenecks the other unnecessarily. The bounded buffers create automatic back-pressure: if saving is slow, its buffer fills, which slows the transform, which slows the download — the pipeline self-regulates to the slowest stage instead of buffering everything in memory.

Benchmark note: Dataflow shines when stages have genuinely different characteristics; it keeps every stage busy at its own optimal rate rather than forcing a single global concurrency level that is wrong for some stages. The BoundedCapacity setting is what prevents memory blow-up — without it, a fast producer feeding a slow consumer buffers unboundedly until you run out of memory. The cost is conceptual complexity: it is more machinery to understand and debug than a simple loop, so reserve it for genuine multi-stage pipelines. For single-stage bounded parallelism, Parallel.ForEachAsync (Way 2) is simpler; for producer/consumer with one stage, Channels (Way 4) are lighter.

---

Way 4: System.Threading.Channels (producer/consumer streaming)

Best used for: Decoupling producers from consumers within a process — one or more producers generate work, one or more consumers process it, connected by an efficient in-memory queue with back-pressure. The modern foundation for in-process streaming and background-work patterns.

The concept: A Channel<T> is a high-performance, thread-safe, async-aware queue. Producers write to it, consumers read from it, and a bounded channel applies back-pressure — when full, writers wait rather than buffering infinitely. It cleanly separates "generating work" from "doing work", each side running at its own pace.

// A bounded channel connecting producers and consumers
var channel = Channel.CreateBounded<WorkItem>(new BoundedChannelOptions(1000)
{
    FullMode = BoundedChannelFullMode.Wait // producers wait when full — back-pressure
});

// PRODUCER(S) — write work as it arrives
_ = Task.Run(async () =>
{
    await foreach (var item in ReadSourceAsync(ct))
        await channel.Writer.WriteAsync(item, ct); // blocks if channel is full
    channel.Writer.Complete(); // signal: no more items
});

// CONSUMER(S) — multiple workers draining the same channel concurrently
var consumers = Enumerable.Range(0, 5).Select(_ => Task.Run(async () =>
{
    // ReadAllAsync yields items until the channel is completed AND drained
    await foreach (var item in channel.Reader.ReadAllAsync(ct))
        await ProcessAsync(item, ct);
})).ToArray();

await Task.WhenAll(consumers); // all work processed

Why it works: The channel is the buffer and the synchronisation mechanism in one — producers and consumers never touch shared mutable state directly, they just write and read the channel, which handles all the thread-safety. The bounded capacity provides back-pressure so a fast producer cannot overwhelm slow consumers and exhaust memory. Multiple consumers on one channel give you a worker pool naturally.

Benchmark note: Channels are extremely efficient — designed for high-throughput scenarios (they are used inside ASP.NET Core itself) with minimal allocation and lock contention. They are the right primitive for the in-process background-queue pattern (see the long-running-operations use case) and any producer/consumer setup. The key choice is bounded vs unbounded: almost always bounded, because an unbounded channel with a fast producer and slow consumers is an out-of-memory crash waiting to happen — the bound is what makes it safe. Channels are simpler than Dataflow when you have a single processing stage; choose Dataflow only when you need multiple linked stages with per-stage tuning.

---

Way 5: SemaphoreSlim & Synchronisation (protect shared resources)

Best used for: Limiting concurrent access to a resource that cannot handle unlimited simultaneous use (a rate-limited API, a connection-limited legacy system), and protecting shared mutable state from race conditions in async code.

The concept: SemaphoreSlim is an async-compatible gate that permits only N concurrent holders. It serves two purposes: as a throttle (allow at most N concurrent operations) and as an async mutual-exclusion primitive (N=1) for protecting shared state across await boundaries — where the lock keyword cannot be used because you cannot await inside a lock.

// THROTTLE — cap concurrent calls to a resource that allows only N at once
public class ThrottledApiClient
{
    private readonly SemaphoreSlim _gate = new(initialCount: 5); // max 5 concurrent

    public async Task<T> CallAsync<T>(Func<Task<T>> operation, CancellationToken ct)
    {
        await _gate.WaitAsync(ct);       // acquire a slot (waits if all 5 are taken)
        try { return await operation(); }
        finally { _gate.Release(); }     // ALWAYS release in finally — or you leak slots
    }
}

// ASYNC MUTUAL EXCLUSION — protect shared state where lock{} cannot be used (it forbids await)
public class SafeCounter
{
    private readonly SemaphoreSlim _mutex = new(1, 1); // N=1 → one at a time
    private int _count;

    public async Task IncrementAsync(CancellationToken ct)
    {
        await _mutex.WaitAsync(ct);
        try
        {
            // Critical section — safe across await. (lock{} would NOT compile with await inside.)
            _count++;
            await _auditLog.RecordAsync(_count, ct);
        }
        finally { _mutex.Release(); }
    }
}

// For simple shared state with NO await, prefer Interlocked (lock-free) or lock{}:
private int _hits;
public void RecordHit() => Interlocked.Increment(ref _hits); // atomic, no locking needed

Why it works: SemaphoreSlim bounds concurrency precisely — exactly N operations proceed, the rest wait their turn — which both protects limited resources and prevents the race conditions that arise when multiple async operations touch shared state simultaneously. Its async WaitAsync integrates with the task model, unlike the synchronous lock keyword which cannot span await points.

Benchmark note: The discipline that matters: always release in a finally block. If an exception skips the release, that permit is leaked forever, and enough leaks deadlock the system as the semaphore permanently runs out of slots. Match the tool to the need: for protecting simple counters or flags with no await, Interlocked (atomic, lock-free, fastest) or the lock keyword is lighter than a semaphore; reach for SemaphoreSlim when you need async waiting or a concurrency limit above 1. Note that .NET 9's new System.Threading.Lock type improves the synchronous lock path but still cannot be held across await — async mutual exclusion remains SemaphoreSlim's job. The cardinal rule of all shared-state concurrency: never mutate shared data from multiple operations without synchronisation, or you get races that are intermittent, environment-dependent, and brutal to debug.

---

Choosing the right approach

• Known small set of independent async calls → Task.WhenAll (Way 1)
• Large collection, bounded concurrency → Parallel.ForEachAsync (Way 2)
• Multi-stage pipeline, per-stage tuning → TPL Dataflow (Way 3)
• Producer/consumer streaming in-process → Channels (Way 4)
• Limit access to a resource / protect shared state → SemaphoreSlim + Interlocked (Way 5)

The decision spine: first, is the work CPU-bound or I/O-bound? I/O-bound work (API/database calls) benefits from high concurrency because it mostly waits — Task.WhenAll for a few, Parallel.ForEachAsync for many. CPU-bound work should not exceed the core count. Second, is it one stage or several? One stage → Channels (producer/consumer) or Parallel.ForEachAsync (collection); multiple stages with different needs → Dataflow. Third, is there shared mutable state? If so, it must be protected (SemaphoreSlim, Interlocked, concurrent collections) — this is where most concurrency bugs live. Two rules dominate everything: bound your concurrency (unbounded parallelism exhausts memory, connections, and downstreams — every approach here has a limit knob, use it) and never share mutable state without synchronisation (the source of races that are maddening to reproduce and fix).

Note: this is an informational engineering overview. Optimal parallelism levels and performance gains depend heavily on workload type, hardware, and downstream limits. Always measure against your own workload.