Long-Running Operations

Use Case: Long-Running Operations (Work That Outlives the Request)

The everyday problem: A user clicks "Generate Report", "Process This Video", or "Import 500,000 Records". The work takes 30 seconds, 5 minutes, or an hour. If you do it inline in the HTTP request, the browser spinner hangs, the request times out (most proxies and load balancers kill connections after 30–120 seconds), the user refreshes and triggers the work again, and a deploy or app-pool recycle mid-request loses everything. The challenge is starting work that takes longer than a reasonable HTTP request should, returning to the user immediately, and letting them track progress and get the result when it is ready — reliably, even across restarts.

Five approaches, from "the tempting mistake everyone makes first" to "durable workflow that survives anything". The right one depends on how long the work runs, whether it must survive a crash, and how the user learns it is done.

---

Way 1: Fire-and-Forget (the tempting mistake — and when it is OK)

Best used for: Genuinely non-critical, fast side-effects where losing the work occasionally is acceptable — bumping a "last seen" timestamp, best-effort cache warming. Almost never the right answer for real work — included here because everyone reaches for it first and needs to understand exactly why it fails.

The concept: You kick off the work on a background thread and return immediately without awaiting it. The request completes; the work continues "somewhere".

// THE MISTAKE — do not do this for important work
[HttpPost("generate-report")]
public IActionResult GenerateReport(ReportRequest request)
{
    // Fire-and-forget — returns instantly, but...
    _ = Task.Run(async () => await _reportService.GenerateAsync(request));
    return Accepted();
}
// WHY THIS IS BROKEN:
// 1. No tracking — the user has no idea if it succeeded, failed, or is still running
// 2. Lost on shutdown — a deploy/recycle kills the thread mid-work, silently
// 3. Unobserved exceptions — a throw inside disappears; you never find out
// 4. No DI scope — the DbContext captured here may be disposed before the task runs
// 5. No back-pressure — 1,000 requests spawn 1,000 threads, exhausting the pool


// SLIGHTLY less broken — register with the host so shutdown waits briefly:
[HttpPost("warm-cache")]
public IActionResult WarmCache([FromServices] IHostApplicationLifetime lifetime)
{
    _ = Task.Run(async () =>
    {
        try { await _cache.WarmAsync(); }
        catch (Exception ex) { _logger.LogError(ex, "Cache warm failed"); } // at least observe it
    });
    return Accepted();
}
// Still loses work on shutdown and has no tracking — only for truly disposable side-effects

Why it (barely) works: It returns instantly and costs nothing to write. That is the entire upside. For anything a user cares about, every one of the five failure modes above will eventually bite you in production.

Benchmark note: There is no performance benefit over the proper approaches — the work still consumes the same CPU. The "cost" is reliability, not speed. The captured-DbContext bug (#4) is the most common: the scoped DbContext is disposed when the request ends, so the background task throws ObjectDisposedException the moment it touches the database — intermittently, under load, impossible to reproduce locally. If you take one thing from this section: fire-and-forget for real work is a latent production incident.

---

Way 2: In-Process Background Queue + Polling (IHostedService)

Best used for: Work that fits on one server, runs seconds to a few minutes, and where a small risk of loss on restart is acceptable — generating a document, sending a batch of emails, processing an upload. The standard "do it properly without new infrastructure" approach.

The concept: A System.Threading.Channels queue holds work items. A BackgroundService reads from the queue and processes items one at a time (or with bounded concurrency) on a long-lived background worker — with its own DI scope. The API enqueues a job, stores its status, and returns a job ID. The client polls a status endpoint.

// 1. A bounded in-memory queue
public class BackgroundTaskQueue
{
    private readonly Channel<Func<IServiceProvider, CancellationToken, Task>> _channel
        = Channel.CreateBounded<Func<IServiceProvider, CancellationToken, Task>>(
            new BoundedChannelOptions(100) { FullMode = BoundedChannelFullMode.Wait });

    public ValueTask EnqueueAsync(Func<IServiceProvider, CancellationToken, Task> work, CancellationToken ct)
        => _channel.Writer.WriteAsync(work, ct); // back-pressure: blocks if queue is full

    public IAsyncEnumerable<Func<IServiceProvider, CancellationToken, Task>> ReadAllAsync(CancellationToken ct)
        => _channel.Reader.ReadAllAsync(ct);
}

// 2. The worker — long-lived, creates a fresh DI scope per job (fixes the fire-and-forget bug)
public class QueuedHostedService(BackgroundTaskQueue queue, IServiceProvider services,
    ILogger<QueuedHostedService> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var work in queue.ReadAllAsync(ct))
        {
            using var scope = services.CreateScope(); // proper scope — DbContext is valid
            try { await work(scope.ServiceProvider, ct); }
            catch (Exception ex) { logger.LogError(ex, "Background job failed"); } // observed
        }
    }
}

// 3. Job status store (in-memory here; use Redis/DB to survive restarts)
public record JobStatus(string Id, string State, int Percent, string? ResultUrl, string? Error);

// 4. The API — enqueue, return a job id, client polls
[HttpPost("reports")]
public async Task<IActionResult> StartReport(ReportRequest req, CancellationToken ct)
{
    var jobId = Guid.NewGuid().ToString();
    _jobs.Set(jobId, new JobStatus(jobId, "Queued", 0, null, null));

    await _queue.EnqueueAsync(async (sp, token) =>
    {
        var svc = sp.GetRequiredService<IReportService>();   // resolved in the worker's scope
        _jobs.Set(jobId, new JobStatus(jobId, "Running", 0, null, null));
        var url = await svc.GenerateAsync(req, p => _jobs.UpdatePercent(jobId, p), token);
        _jobs.Set(jobId, new JobStatus(jobId, "Completed", 100, url, null));
    }, ct);

    return Accepted(new { jobId, statusUrl = $"/reports/{jobId}/status" });
}

[HttpGet("reports/{jobId}/status")]
public IActionResult GetStatus(string jobId)
    => _jobs.TryGet(jobId, out var status) ? Ok(status) : NotFound();

Why it works: The request returns instantly with a job ID. The worker processes jobs on its own thread with a proper DI scope, so the DbContext is valid and exceptions are logged. The bounded channel provides back-pressure — under flood, enqueue waits instead of spawning unbounded threads. The client polls a clean status endpoint.

Benchmark note: This handles hundreds of jobs per minute comfortably on a single instance with predictable memory (bounded queue). The defining limitation is the word in-process: jobs live in this server's memory, so a deploy or crash loses queued and in-flight work, and it does not distribute across multiple instances (each server has its own queue). Register a graceful-shutdown drain (await in-flight jobs in StopAsync with a timeout) to reduce — not eliminate — loss on planned restarts. When you need durability or multi-server distribution, move to Way 3.

---

Way 3: Durable Job Queue (Hangfire / Azure Queue / SQS + worker)

Best used for: Work that must survive restarts and crashes, must distribute across multiple servers, needs automatic retries, and may run minutes to hours — the production default for serious background processing.

The concept: Jobs are persisted to durable storage (a database table via Hangfire, or a cloud queue like Azure Storage Queues / AWS SQS) before the request returns. Separate worker processes (which can scale independently of your web servers) pull jobs from the store, process them, and mark them complete. If a worker dies mid-job, the job becomes visible again and another worker picks it up.

// --- Option A: Hangfire (persists jobs to SQL Server / PostgreSQL / Redis) ---
// Program.cs
builder.Services.AddHangfire(c => c.UsePostgreSqlStorage(connStr));
builder.Services.AddHangfireServer(); // runs workers in-process, or host separately

// Enqueue — persisted to the database before returning
[HttpPost("reports")]
public IActionResult StartReport(ReportRequest req)
{
    // Survives restarts: the job row is committed to the DB immediately
    string jobId = BackgroundJob.Enqueue<IReportService>(
        svc => svc.GenerateAsync(req, CancellationToken.None));
    return Accepted(new { jobId });
}

// Hangfire gives retries, scheduling, and a dashboard for free:
BackgroundJob.Schedule<IEmailService>(s => s.SendReminderAsync(userId), TimeSpan.FromHours(24));
RecurringJob.AddOrUpdate<ICleanupService>("nightly", s => s.PurgeAsync(), Cron.Daily);


// --- Option B: Cloud queue (Azure Storage Queue) + a worker service ---
// Producer (API): enqueue a message, return immediately
[HttpPost("reports")]
public async Task<IActionResult> StartReport(ReportRequest req, CancellationToken ct)
{
    var jobId = Guid.NewGuid().ToString();
    await _dbJobs.CreateAsync(jobId, "Queued", ct);            // status row in DB
    var msg = JsonSerializer.Serialize(new { jobId, req });
    await _queueClient.SendMessageAsync(msg, ct);             // durable cloud queue
    return Accepted(new { jobId, statusUrl = $"/reports/{jobId}/status" });
}

// Consumer (separate worker / Azure Function / hosted service):
public class ReportWorker(QueueClient queue, IServiceProvider services) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var messages = await queue.ReceiveMessagesAsync(maxMessages: 16, cancellationToken: ct);
            foreach (var m in messages.Value)
            {
                using var scope = services.CreateScope();
                try
                {
                    var job = JsonSerializer.Deserialize<ReportJob>(m.MessageText)!;
                    await scope.ServiceProvider.GetRequiredService<IReportService>()
                        .GenerateAsync(job.Req, ct);
                    await queue.DeleteMessageAsync(m.MessageId, m.PopReceipt, ct); // ack — remove
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Job failed; message will reappear for retry");
                    // NOT deleting the message → it becomes visible again after the timeout
                    // After N failed attempts the queue moves it to a poison/dead-letter queue
                }
            }
        }
    }
}

Why it works: The job is durably persisted before the request returns, so a crash loses nothing — the job is still in the store. Workers are separate from web servers, so you scale processing independently of traffic. The "delete on success, leave on failure" pattern gives automatic retry and dead-lettering: a job that keeps failing eventually lands in a poison queue for inspection rather than blocking the line forever.

Benchmark note: Throughput scales horizontally — add worker instances to process more jobs in parallel. Hangfire is the lowest-friction option (one NuGet package, uses your existing database, includes a dashboard) and comfortably handles thousands of jobs/day; its storage is your relational DB, so extremely high volumes (millions/day) eventually pressure the database and argue for a dedicated queue. Cloud queues (SQS/Azure) scale to effectively unlimited throughput but you build the status tracking and dashboard yourself. The key reliability rule: make jobs idempotent (see the idempotency use case) because at-least-once delivery means a job can run twice if a worker crashes after finishing but before acknowledging.

---

Way 4: SignalR Push (tell the client the moment it is done)

Best used for: Interactive apps where the user is waiting and watching — live progress bars, "your export is ready" notifications, dashboards — and you want to push the result instantly instead of making the client poll.

The concept: Combine a durable queue (Way 3) for the actual processing with a SignalR connection for notification. The worker, as it progresses and completes, pushes updates over a persistent WebSocket connection to the specific user. No polling — the server tells the client the instant something changes.

// SignalR hub
public class JobHub : Hub
{
    // Client joins a group named after their job so we can target it
    public Task SubscribeToJob(string jobId) => Groups.AddToGroupAsync(Context.ConnectionId, jobId);
}

// The worker pushes progress and completion — no polling needed
public class ReportWorker(IHubContext<JobHub> hub, IServiceProvider services) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var job in DequeueJobsAsync(ct))
        {
            using var scope = services.CreateScope();
            var svc = scope.ServiceProvider.GetRequiredService<IReportService>();

            var url = await svc.GenerateAsync(job.Req, async percent =>
            {
                // Push live progress to just this job's subscribers
                await hub.Clients.Group(job.Id).SendAsync("Progress", new { job.Id, percent }, ct);
            }, ct);

            // Push completion the instant it finishes
            await hub.Clients.Group(job.Id).SendAsync("Completed", new { job.Id, url }, ct);
        }
    }
}

// Client (JavaScript) — subscribe and react to pushes
const conn = new signalR.HubConnectionBuilder().withUrl("/jobhub").withAutomaticReconnect().build();
conn.on("Progress",  d => updateProgressBar(d.percent));
conn.on("Completed", d => showDownloadLink(d.url));
await conn.start();
await conn.invoke("SubscribeToJob", jobId);

Why it works: The persistent connection eliminates polling entirely — the client learns of progress and completion within milliseconds of it happening, and you avoid the constant "are we there yet?" status requests that polling generates. The heavy work still runs on the durable queue; SignalR is purely the notification channel.

Benchmark note: Polling every 2 seconds for a 5-minute job is 150 wasted requests per client; SignalR replaces all of them with a few push messages. The trade-off is connection state: SignalR connections consume server memory and, across multiple servers, require a backplane (Redis) so the worker on server B can push to a client connected to server A. SignalR 9's stateful reconnect helps clients survive brief drops without missing the completion message. Use this when the UX of instant feedback justifies the connection management — for fire-once batch jobs where nobody is watching, polling (Way 3) is simpler.

---

Way 5: Durable Workflow (Azure Durable Functions / Temporal / workflow engine)

Best used for: Multi-step processes that span minutes to days, involve waiting for external events or human approval, must survive any failure at any step, and need to resume exactly where they left off — order fulfilment, document approval chains, multi-stage data pipelines, anything with "wait for X then do Y".

The concept: A durable workflow engine persists the state of the orchestration itself after every step. The workflow code reads like a normal sequential method — call step 1, await it, call step 2 — but the engine checkpoints progress so that if the process crashes at step 7 of 10, it resumes at step 7 on restart, not from the beginning. It can also pause for external events (an approval, a webhook, a timer) for arbitrarily long without holding any thread or connection.

// Azure Durable Functions — orchestration that survives crashes and waits
[Function("ProcessOrderOrchestrator")]
public async Task RunOrchestrator([OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<Order>()!;

    // Each CallActivityAsync is checkpointed — a crash resumes at the last completed step
    await context.CallActivityAsync("ReserveInventory", order);
    await context.CallActivityAsync("ChargePayment", order);

    // Wait for an EXTERNAL event — e.g. a human approval — for up to 3 days,
    // holding NO thread and NO connection while waiting:
    using var cts = new CancellationTokenSource();
    var approval = context.WaitForExternalEvent<bool>("ManagerApproval");
    var timeout  = context.CreateTimer(context.CurrentUtcDateTime.AddDays(3), cts.Token);

    if (await Task.WhenAny(approval, timeout) == approval)
    {
        cts.Cancel();
        await context.CallActivityAsync("ShipOrder", order);
    }
    else
    {
        // Timed out — compensate (the SAGA pattern; see distributed-transactions use case)
        await context.CallActivityAsync("RefundPayment", order);
        await context.CallActivityAsync("ReleaseInventory", order);
    }
}

// An activity is a normal, idempotent unit of work
[Function("ChargePayment")]
public async Task ChargePayment([ActivityTrigger] Order order)
    => await _payments.ChargeAsync(order);

// Start the workflow from an API and hand back a status-query URL
[Function("StartOrder")]
public async Task<HttpResponseData> Start(
    [HttpTrigger] HttpRequestData req,
    [DurableClient] DurableTaskClient client)
{
    var order = await req.ReadFromJsonAsync<Order>();
    string instanceId = await client.ScheduleNewOrchestrationInstanceAsync("ProcessOrderOrchestrator", order);
    return client.CreateCheckStatusResponse(req, instanceId); // built-in status endpoints
}

Why it works: The engine persists orchestration state after every awaited step (event sourcing under the hood), so the workflow is crash-proof and can pause indefinitely without consuming resources. The code stays readable as a linear sequence even though it may execute across days, multiple machines, and several restarts. Built-in support for timers, external events, retries, and compensation makes complex long-running business processes tractable.

Benchmark note: This is about reliability and duration, not throughput — the engine adds per-step persistence overhead (each activity call is a durable write), so it is the wrong tool for high-frequency short tasks (use Way 3 for those). Its sweet spot is processes measured in minutes to days where correctness across failures is paramount. Temporal (self-hosted/cloud) offers the same model language-agnostically; Azure Durable Functions is the native .NET option on Azure. The main caveat is the orchestrator constraint: orchestrator code must be deterministic (no DateTime.Now, no direct I/O, no random) because it is replayed during recovery — all non-deterministic work goes in activities.

---

Choosing the right approach

• Disposable side-effect, loss acceptable → Fire-and-forget, observed (Way 1) — and even then, reluctantly
• Single server, seconds–minutes, no new infra → In-process queue + polling (Way 2)
• Must survive crashes, scale across servers, retries → Durable queue / Hangfire (Way 3) — the production default
• User is watching, want instant feedback → Way 3 for processing + SignalR push (Way 4)
• Multi-step, waits for events/approval, spans days → Durable workflow (Way 5)

The decision spine: the moment work outlives a comfortable request (say, >5–10 seconds), get it off the request thread. If losing it on restart is unacceptable — and it usually is — it must be durably persisted before you return (Way 3+), which immediately means jobs must be idempotent because durable queues deliver at-least-once. Layer SignalR on top when the user is waiting (Way 4), and graduate to a workflow engine (Way 5) only when the process has multiple steps that must individually survive failure or wait for external events. The recurring theme across all of these: return fast, process durably, track status, make it idempotent.

Note: this is an informational engineering overview. Performance and reliability characteristics depend on infrastructure, configuration, and workload. Always validate against your own environment.