Background & Scheduled Jobs

Use Case: Background & Scheduled Jobs

The everyday problem: Work needs to happen outside a user request — send a nightly digest email at 2 AM, retry failed payments every 15 minutes, clean up expired sessions hourly, generate monthly invoices on the 1st, poll a partner API every minute, process items from a queue continuously. None of this is triggered by a user clicking a button; it runs on a schedule or continuously in the background. The challenge is running recurring and deferred work reliably — surviving restarts and deploys, not running twice when you have multiple servers, recovering from failures, and being observable when something goes wrong.

Five approaches, from the built-in BackgroundService to cloud-native schedulers. The key differentiators are durability (does a missed run get caught up?), distribution (what happens with multiple instances?), and visibility (can you see what ran and what failed?).

---

Way 1: BackgroundService / IHostedService (built-in, continuous)

Best used for: Continuous background loops and simple timers that live with your app — polling a queue, periodic cache refresh, a heartbeat — where the work is in-process and you do not need durable scheduling history.

The concept: BackgroundService is the framework's base class for long-running background work. You override ExecuteAsync with a loop that does work and waits. It starts when the app starts, stops gracefully on shutdown, and runs in-process alongside your web server (or in a dedicated worker service).

// A periodic background job using the modern PeriodicTimer
public class SessionCleanupService(IServiceProvider services, ILogger<SessionCleanupService> logger)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // PeriodicTimer — no timer drift, fully async, respects cancellation
        using var timer = new PeriodicTimer(TimeSpan.FromHours(1));

        while (await timer.WaitForNextTickAsync(ct))
        {
            using var scope = services.CreateScope(); // fresh DI scope each run — valid DbContext
            try
            {
                var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
                int removed = await db.Sessions
                    .Where(s => s.ExpiresAt < DateTime.UtcNow)
                    .ExecuteDeleteAsync(ct);
                logger.LogInformation("Cleaned up {Count} expired sessions", removed);
            }
            catch (OperationCanceledException) { break; } // graceful shutdown
            catch (Exception ex)
            {
                logger.LogError(ex, "Session cleanup failed; will retry next tick");
                // Swallow so one failure does not kill the whole service
            }
        }
    }
}

builder.Services.AddHostedService<SessionCleanupService>();

Why it works: It is built into the framework — no dependencies — and integrates with the host lifecycle, so it starts and stops cleanly with the app. PeriodicTimer (modern .NET) avoids the re-entrancy and drift problems of older System.Threading.Timer callbacks, and creating a DI scope per iteration gives each run a valid DbContext.

Benchmark note: Zero infrastructure and minimal overhead — ideal for continuous loops and simple periodic tasks. The limitations define when to graduate: it is in-process and non-durable, so a missed run (app was down at the scheduled time) is simply skipped, not caught up; there is no history of past runs to inspect; and with multiple instances it runs on every instance — five servers means five simultaneous cleanups, which is wasteful at best and corrupting at worst (five workers grabbing the same queue items). For a single instance or genuinely idempotent per-instance work it is perfect; beyond that you need distributed coordination (Ways 3–5).

---

Way 2: Cron-Style Scheduling (Quartz.NET)

Best used for: Complex schedules that PeriodicTimer cannot express — "every weekday at 9 AM", "the last Friday of each month", "every 15 minutes between 8 AM and 6 PM" — with persistence and clustering so jobs survive restarts and do not double-run across instances.

The concept: Quartz.NET is a full scheduling library: jobs (the work) are decoupled from triggers (when to run, expressed as cron expressions or calendars). With a persistent job store (database) and clustering enabled, it coordinates across multiple instances so each job fires exactly once cluster-wide, and survives restarts.

builder.Services.AddQuartz(q =>
{
    var jobKey = new JobKey("nightly-invoices");
    q.AddJob<GenerateInvoicesJob>(jobKey);
    q.AddTrigger(t => t
        .ForJob(jobKey)
        .WithCronSchedule("0 0 2 1 * ?")); // 02:00 on the 1st of every month

    q.AddTrigger(t => t
        .ForJob("retry-payments")
        .WithCronSchedule("0 0/15 8-18 ? * MON-FRI")); // every 15m, 8am-6pm, weekdays

    // Persistence + clustering: jobs survive restarts and fire once across the cluster
    q.UsePersistentStore(s =>
    {
        s.UsePostgres(connStr);
        s.UseClustering();         // multiple instances coordinate via the database
        s.UseSystemTextJsonSerializer();
    });
});
builder.Services.AddQuartzHostedService(o => o.WaitForJobsToComplete = true);

[DisallowConcurrentExecution] // never run two instances of THIS job at once
public class GenerateInvoicesJob(IServiceProvider services) : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        using var scope = services.CreateScope();
        await scope.ServiceProvider.GetRequiredService<IInvoiceService>()
            .GenerateMonthlyAsync(context.CancellationToken);
    }
}

Why it works: Cron expressions handle arbitrarily complex schedules declaratively. The persistent store means a job scheduled for 2 AM still fires even if the app restarted at 1:59; clustering means the database acts as the coordinator so exactly one instance runs each fire, solving the "runs on every server" problem of Way 1. [DisallowConcurrentExecution] prevents overlapping runs of a slow job.

Benchmark note: Quartz adds a database-backed coordination layer — a small cost per scheduled fire (acquiring a cluster lock) in exchange for correct exactly-once cluster-wide scheduling and durability. It is the right choice when schedules are complex or when you need clustering without adopting a heavier system. The trade-off versus Way 3 is observability: Quartz has no built-in dashboard (you query its tables or add one), whereas Hangfire ships with a rich UI. For pure scheduling with complex cron needs and existing operational maturity, Quartz is excellent; for ad-hoc background jobs with a dashboard, many teams prefer Hangfire.

---

Way 3: Hangfire (durable jobs + dashboard)

Best used for: A mix of fire-and-forget background jobs, delayed jobs, and recurring jobs — with automatic retries, persistence, and a built-in dashboard — when you want the most operational visibility for the least setup.

The concept: Hangfire persists all jobs (enqueued, scheduled, recurring) to a database. Worker threads pick them up and execute them, retrying failures automatically with backoff. Its standout feature is the dashboard: a web UI showing every job's state, history, failures, and retries — turning background processing from a black box into something observable.

builder.Services.AddHangfire(c => c.UsePostgreSqlStorage(connStr));
builder.Services.AddHangfireServer();
// app.UseHangfireDashboard("/hangfire"); // the built-in monitoring UI

// Recurring job — declared once, runs on schedule, survives restarts, fires once cluster-wide
RecurringJob.AddOrUpdate<IDigestService>(
    "daily-digest",
    svc => svc.SendDailyDigestAsync(CancellationToken.None),
    "0 8 * * *");  // 8 AM daily (cron)

// Delayed job — run once, later
BackgroundJob.Schedule<IReminderService>(
    svc => svc.SendTrialEndingReminderAsync(userId),
    TimeSpan.FromDays(7));

// Fire-and-forget but DURABLE — persisted before returning, auto-retried on failure
BackgroundJob.Enqueue<IEmailService>(svc => svc.SendWelcomeAsync(userId));

// Continuations — run B after A succeeds
var jobId = BackgroundJob.Enqueue<IImportService>(s => s.ImportAsync(fileId));
BackgroundJob.ContinueJobWith<INotifyService>(jobId, s => s.NotifyImportDoneAsync(fileId));

Why it works: Everything is persisted, so nothing is lost on restart and recurring jobs catch up on missed schedules. Automatic retry with backoff handles transient failures without custom code. The dashboard makes the entire system observable — you can see what ran, what failed, why, and retry it manually. It coordinates across instances so recurring jobs fire once cluster-wide.

Benchmark note: Hangfire's storage is your relational database, which is also its scaling ceiling: it comfortably handles thousands to tens of thousands of jobs per day, but very high volumes (millions/day) put real load on the database (job state polling, history) and argue for a dedicated queue system instead. For the overwhelming majority of applications that mix scheduled and background work, the durability-plus-dashboard combination is the sweet spot — minimal setup, maximum visibility. Keep jobs idempotent (retries mean a job can run more than once) and configure history retention so the tables do not grow forever.

---

Way 4: Cloud Scheduler + Queue Trigger (serverless)

Best used for: Cloud-native architectures (Azure, AWS) where you want managed scheduling with no servers to maintain — the scheduler and the compute are platform services that scale to zero when idle.

The concept: The cloud platform provides the schedule trigger (Azure Functions timer trigger, AWS EventBridge Scheduler) and the execution environment (Functions / Lambda). You write just the job logic; the platform handles when it runs, scaling, and availability. Often combined with queue triggers so a timer enqueues work that a separate function processes — decoupling scheduling from heavy processing.

// Azure Functions — timer trigger (the schedule is managed by the platform)
public class ScheduledFunctions(IInvoiceService invoices, QueueClient queue)
{
    // Runs on a cron schedule managed by Azure — no server, scales automatically
    [Function("EnqueueMonthlyInvoices")]
    public async Task EnqueueInvoices(
        [TimerTrigger("0 0 2 1 * *")] TimerInfo timer, // 2 AM on the 1st
        CancellationToken ct)
    {
        // Keep the timer fast: just enqueue work, do not process it inline
        var customerIds = await invoices.GetCustomersDueAsync(ct);
        foreach (var id in customerIds)
            await queue.SendMessageAsync(id.ToString(), ct);
    }

    // Queue trigger — scales out automatically to process the enqueued work in parallel
    [Function("GenerateInvoice")]
    public async Task GenerateInvoice(
        [QueueTrigger("invoice-jobs")] string customerId,
        CancellationToken ct)
    {
        await invoices.GenerateForCustomerAsync(int.Parse(customerId), ct);
        // Platform auto-retries on failure and dead-letters poison messages
    }
}

Why it works: The platform owns scheduling, scaling, and availability — there is no always-on server to pay for or patch, and it scales to zero when idle. The timer-enqueues-then-queue-processes split means the scheduled trigger stays fast and reliable while the actual work scales out across many parallel function executions, each processing one item.

Benchmark note: Cost-efficient for spiky or infrequent workloads (you pay per execution, nothing when idle) and effortlessly scalable (the platform fans out queue processing automatically). The trade-offs are platform lock-in (the triggers are Azure/AWS-specific), cold starts (the first invocation after idle has added latency — usually irrelevant for scheduled jobs but worth knowing), and execution time limits (Functions/Lambda cap how long a single execution can run, so very long jobs must be broken into queued chunks). For cloud-hosted systems with variable background workloads, this is often the most economical and operationally simplest option.

---

Way 5: Distributed Lock for Single-Instance Jobs (coordinate Way 1)

Best used for: When you want to keep the simplicity of BackgroundService (Way 1) but run multiple app instances and need a scheduled job to execute on only one of them — the lightweight fix for the "runs on every server" problem without adopting Quartz or Hangfire.

The concept: All instances run the same BackgroundService, but before doing the work, each tries to acquire a distributed lock (in Redis or the database). Only the one that wins the lock proceeds; the others skip this tick. The lock has a TTL so that if the winner crashes mid-job, the lock expires and another instance can take over next time.

public class DistributedScheduledJob(IServiceProvider services, IConnectionMultiplexer redis,
    ILogger<DistributedScheduledJob> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(15));
        var db = redis.GetDatabase();

        while (await timer.WaitForNextTickAsync(ct))
        {
            var lockKey   = "job:retry-payments:lock";
            var lockValue = Environment.MachineName + Guid.NewGuid(); // unique owner token
            var lockTtl   = TimeSpan.FromMinutes(10); // must exceed worst-case job duration

            // SET key value NX EX ttl — atomic acquire; only ONE instance succeeds
            bool acquired = await db.StringSetAsync(lockKey, lockValue, lockTtl, When.NotExists);
            if (!acquired)
            {
                logger.LogDebug("Another instance holds the lock; skipping this tick");
                continue;
            }

            try
            {
                using var scope = services.CreateScope();
                await scope.ServiceProvider.GetRequiredService<IPaymentRetryService>()
                    .RetryFailedAsync(ct);
            }
            finally
            {
                // Release only if WE still own it (avoid releasing a lock that already expired
                // and was re-acquired by someone else) — done safely via a Lua compare-and-delete
                const string release = @"if redis.call('GET', KEYS[1]) == ARGV[1]
                                         then return redis.call('DEL', KEYS[1]) else return 0 end";
                await db.ScriptEvaluateAsync(release, new RedisKey[] { lockKey },
                    new RedisValue[] { lockValue });
            }
        }
    }
}

Why it works: The atomic SET ... NX (set-if-not-exists) means exactly one instance acquires the lock per tick; the rest see it is held and skip. The TTL provides crash recovery — if the holder dies, the lock auto-expires and another instance picks up the next cycle. The compare-and-delete on release ensures an instance only releases its own lock, never one that expired and was re-acquired by another.

Benchmark note: This keeps Way 1's zero-infrastructure simplicity (just a BackgroundService) while adding correct single-execution across a farm for the cost of one Redis round-trip per tick. The critical tuning parameter is the lock TTL: it must comfortably exceed the job's worst-case runtime, or the lock expires mid-job and a second instance starts running concurrently. For jobs whose duration is unpredictable, this is fragile — Quartz/Hangfire (Ways 2–3) handle long jobs and coordination more robustly. But for short, well-bounded periodic jobs in a multi-instance app, the distributed-lock pattern is a pragmatic, dependency-light solution. (Libraries like DistributedLock wrap this correctly across Redis, SQL Server, and Postgres.)

---

Choosing the right approach

• Continuous loop / simple timer, single instance → BackgroundService + PeriodicTimer (Way 1)
• Complex cron schedules, need clustering + persistence → Quartz.NET (Way 2)
• Mixed background/scheduled jobs, want a dashboard + retries → Hangfire (Way 3)
• Cloud-native, no servers, pay-per-run → Cloud scheduler + queue trigger (Way 4)
• Keep BackgroundService but run on one instance only → Distributed lock (Way 5)

The decision spine: start with three questions. Single instance or many? — a single instance can use BackgroundService (Way 1) freely; multiple instances need coordination (Ways 2, 3, or 5) or the job runs N times. How complex is the schedule? — simple intervals suit PeriodicTimer; real cron expressions want Quartz or Hangfire. How much do you need to see? — if observability of past runs and failures matters (it usually does in production), Hangfire's dashboard is the strongest argument for it. Across all of them, two rules hold: create a fresh DI scope per run so your DbContext is valid, and make jobs idempotent because durable schedulers can fire a job more than once after a crash. The progression most teams follow is BackgroundService → distributed lock when they scale out → Hangfire/Quartz when they need durability, history, and complex schedules.

Note: this is an informational engineering overview. Behaviour and limits depend on infrastructure, configuration, and workload. Always validate scheduling, coordination, and failure recovery against your own environment.