File & Blob Handling at Scale

Use Case: File & Blob Handling at Scale

The everyday problem: Users upload profile pictures, documents, videos; your app generates reports and exports; you serve images and downloads. Done naively — receiving the whole file into memory, storing it in the database, streaming it back through your application — files quietly destroy a service: a few hundred concurrent 50 MB uploads exhaust memory, large files in the database bloat it and slow every query, and routing every download through your app servers wastes their bandwidth and capacity. The challenge is handling files — upload, store, serve — at scale, without your application servers becoming the bottleneck for what is essentially bulk byte movement.

Five approaches, the unifying theme being: keep large files out of your application's memory and off its critical path, delegating byte movement to purpose-built storage and delivery infrastructure.

---

Way 1: Streaming Upload (never buffer the whole file)

Best used for: Accepting file uploads of any meaningful size — the baseline correct way to receive files, replacing the default model binding that buffers the entire file into memory.

The concept: By default, ASP.NET Core model binding reads an uploaded file fully into memory (or a temp file) before your code runs. For large files this spikes memory. Streaming instead reads the request body in chunks and writes each chunk straight to its destination (disk, blob storage), so only a small buffer is ever in memory regardless of file size.

// Stream the upload directly to storage — constant memory regardless of file size
[HttpPost("upload")]
[DisableRequestSizeLimit] // or set a sensible explicit limit
public async Task<IActionResult> Upload(CancellationToken ct)
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
        return BadRequest("Expected multipart/form-data.");

    var boundary = MultipartRequestHelper.GetBoundary(Request.ContentType);
    var reader   = new MultipartReader(boundary, Request.Body);

    var section = await reader.ReadNextSectionAsync(ct);
    while (section is not null)
    {
        if (ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var cd)
            && cd.DispositionType.Equals("form-data") && cd.FileName.HasValue)
        {
            // Stream the section body straight to blob storage — no full-file buffer
            var blobClient = _containerClient.GetBlobClient(SafeName(cd.FileName.Value));
            await blobClient.UploadAsync(section.Body, overwrite: true, ct);
            // section.Body is read in chunks; memory stays flat even for a 5 GB file
        }
        section = await reader.ReadNextSectionAsync(ct);
    }
    return Ok();
}

// Disable form-value buffering for streaming endpoints
[GenerateAntiforgeryTokenAttribute]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter { /* ... */ }

Why it works: Reading and writing in chunks means the file flows through a small fixed-size buffer rather than being held whole in memory. A 5 GB upload uses the same few kilobytes of buffer as a 5 KB one, so concurrent large uploads no longer threaten to exhaust memory.

Benchmark note: The difference is categorical: buffered uploads consume memory proportional to (file size × concurrent uploads) — 100 concurrent 50 MB uploads is ~5 GB of memory, enough to crash a typical instance — while streaming holds essentially constant memory regardless. Always set a sensible maximum request size even on streaming endpoints (an unbounded upload is a denial-of-service vector). Validate the file (type, size, content) as you stream, and generate safe storage names rather than trusting the client's filename (which can contain path-traversal sequences). This is the mirror image of the streaming read from the large-data use case.

---

Way 2: Presigned URLs / Direct-to-Storage Upload (bypass your servers entirely)

Best used for: High-volume or large-file uploads where you want the file to never touch your application servers at all — user media, large documents, video — letting clients upload straight to cloud storage.

The concept: Instead of the file passing through your server on its way to storage, your server issues a short-lived, scoped presigned URL (a temporary credential granting permission to upload one specific blob). The client uploads directly to Azure Blob Storage / S3 using that URL. Your server never handles the bytes — it only authorises the operation.

// Server: issue a short-lived, single-purpose upload URL — no bytes pass through here
[HttpPost("uploads/presign")]
public IActionResult PresignUpload(string fileName, CancellationToken ct)
{
    var blobName   = $"{Guid.NewGuid()}/{SafeName(fileName)}";
    var blobClient = _containerClient.GetBlobClient(blobName);

    // SAS token granting WRITE to this ONE blob, valid for 15 minutes only
    var sas = blobClient.GenerateSasUri(new BlobSasBuilder
    {
        BlobContainerName = _containerClient.Name,
        BlobName          = blobName,
        Resource          = "b",
        ExpiresOn         = DateTimeOffset.UtcNow.AddMinutes(15)
    }.SetPermissions(BlobSasPermissions.Write));

    // Client will PUT the file directly to this URL
    return Ok(new { uploadUrl = sas.ToString(), blobName });
}

// AWS S3 equivalent: GetPreSignedUrl with PUT verb and short expiry

// Client (JavaScript): upload straight to storage — never touches the app server
async function upload(file) {
  const { uploadUrl, blobName } = await fetch('/uploads/presign?fileName=' + file.name,
      { method: 'POST' }).then(r => r.json());

  await fetch(uploadUrl, {                          // direct to Azure/S3
    method: 'PUT',
    headers: { 'x-ms-blob-type': 'BlockBlob' },
    body: file
  });

  await fetch('/uploads/complete', {               // tell the server it's done
    method: 'POST', body: JSON.stringify({ blobName })
  });
}

Why it works: Your application servers are removed from the data path entirely — they spend a millisecond issuing a URL instead of minutes shuffling gigabytes. The cloud storage service, built for exactly this and effectively infinitely scalable, handles the actual transfer. Your server capacity is freed for application logic, and upload throughput is limited by the storage service (enormous) rather than your servers (finite).

Benchmark note: This is transformative for file-heavy applications: upload capacity becomes the cloud provider's near-unlimited bandwidth instead of your servers', and your instances stay small and cheap because they never move file bytes. The trade-offs are coordination and validation: the client makes two round-trips (presign, then a completion notification), and since the file bypassed your server, you validate after upload (check size/type from blob metadata, scan for malware) — often via an event the storage service fires on upload completion. Scope the presigned URL tightly (one blob, write-only, short expiry) so it cannot be abused. For anything beyond modest files at modest volume, this is the right model.

---

Way 3: CDN Delivery for Downloads (serve from the edge)

Best used for: Serving files to many users — images, static assets, public documents, media — where routing downloads through your application servers wastes their bandwidth and adds latency for distant users.

The concept: A Content Delivery Network (CDN) caches your files at edge locations distributed worldwide. Users download from the nearest edge node, not from your origin servers. Your servers (or blob storage) are hit once per file per edge cache-fill; thereafter the CDN serves it, offloading the bandwidth and bringing the file geographically closer to each user.

// Generate a CDN URL (optionally with a signed token for access control)
public string GetDownloadUrl(string blobName, bool isPublic)
{
    if (isPublic)
        return $"https://cdn.myapp.com/{blobName}"; // served from the edge, cached

    // For private files: a short-lived signed CDN/SAS URL
    var blobClient = _containerClient.GetBlobClient(blobName);
    var sas = blobClient.GenerateSasUri(new BlobSasBuilder
    {
        BlobContainerName = _containerClient.Name,
        BlobName          = blobName,
        Resource          = "b",
        ExpiresOn         = DateTimeOffset.UtcNow.AddHours(1)
    }.SetPermissions(BlobSasPermissions.Read));
    return sas.ToString();
}

// Set cache headers on the blob so the CDN caches it appropriately
await blobClient.SetHttpHeadersAsync(new BlobHttpHeaders
{
    CacheControl = "public, max-age=86400", // CDN + browser cache for 24h
    ContentType  = "image/jpeg"
});

Why it works: The CDN's edge nodes absorb the download traffic that would otherwise hit your origin, and serve each user from a nearby location — lower latency and massive bandwidth offload. A file requested a million times is fetched from your origin only a handful of times (once per edge node per cache lifetime); the CDN serves the rest.

Benchmark note: CDN delivery cuts origin bandwidth dramatically (often 90%+ offload for popular content) and reduces latency for geographically distributed users from hundreds of milliseconds to tens. It is essential for any public media at scale and effectively mandatory for global audiences. The considerations: cache invalidation (when a file changes, you must purge the CDN or use versioned URLs like image-v2.jpg so users do not get stale content — the same invalidation challenge as caching generally), and access control for private files (use short-lived signed URLs so the CDN can serve protected content without exposing it permanently). Set Cache-Control headers deliberately to control caching behaviour.

---

Way 4: Chunked / Resumable Uploads (survive interruels on large files)

Best used for: Very large files over unreliable connections — video uploads, large datasets, mobile uploads — where a single failed transfer of a multi-gigabyte file (and starting over) is unacceptable.

The concept: Split the file into chunks (say 5 MB each), upload each chunk independently, and assemble them server-side once all arrive. If the connection drops, only the in-flight chunk is lost — the upload resumes from the last successful chunk rather than restarting. Cloud storage supports this natively (Azure block blobs, S3 multipart upload).

// Azure Block Blob: upload in independent blocks, then commit the block list
public async Task UploadChunkAsync(string blobName, int blockIndex, Stream chunk, CancellationToken ct)
{
    var blobClient = _containerClient.GetBlockBlobClient(blobName);
    // Each block has an ID; blocks can be uploaded in any order, retried independently
    var blockId = Convert.ToBase64String(BitConverter.GetBytes(blockIndex));
    await blobClient.StageBlockAsync(blockId, chunk, cancellationToken: ct);
    // A dropped connection loses only THIS block — client retries just this one
}

// After all chunks are staged, commit them in order into the final blob
public async Task CompleteUploadAsync(string blobName, int totalBlocks, CancellationToken ct)
{
    var blobClient = _containerClient.GetBlockBlobClient(blobName);
    var blockIds = Enumerable.Range(0, totalBlocks)
        .Select(i => Convert.ToBase64String(BitConverter.GetBytes(i)));
    await blobClient.CommitBlockListAsync(blockIds, cancellationToken: ct);
    // The staged blocks become one assembled blob
}

// The client tracks which chunks succeeded and re-sends only failed/missing ones on resume.
// S3 equivalent: CreateMultipartUpload → UploadPart (per chunk) → CompleteMultipartUpload

Why it works: Independence of chunks is the key: each chunk is a separate, retryable operation, so a network failure costs only the current chunk (a few megabytes) rather than the entire transfer. The client tracks progress and resumes from where it stopped. This turns "upload a 5 GB file over flaky mobile data" from near-impossible into reliable.

Benchmark note: For large files on unreliable networks, resumability is the difference between uploads that succeed and uploads that perpetually fail and restart. Chunking also enables parallel chunk upload (multiple chunks in flight at once) for higher throughput, and real progress reporting (chunks completed / total). The cost is client and server complexity — tracking chunk state, handling assembly, cleaning up abandoned partial uploads (set a lifecycle policy to delete uncommitted blocks after N days). Libraries and protocols (tus.io for resumable uploads) standardise this. Use it when files are large enough and networks unreliable enough that whole-file uploads genuinely fail; for small files on good connections it is unnecessary overhead.

---

Way 5: Tiered Storage & Lifecycle Management (right-cost each file)

Best used for: Controlling storage cost as data accumulates — keeping frequently-accessed files on fast storage while automatically moving old, rarely-touched files to cheaper tiers, and deleting what is no longer needed.

The concept: Cloud storage offers tiers at different price/performance points — hot (frequent access, higher storage cost, low access cost), cool (infrequent, lower storage cost), and archive (rare, cheapest storage, slow and costly retrieval). Lifecycle policies automatically transition files between tiers based on age or access patterns, and delete expired data — so each file costs what is appropriate for how it is actually used.

// Set a blob's access tier explicitly
await blobClient.SetAccessTierAsync(AccessTier.Cool, cancellationToken: ct);

// Lifecycle management policy (configured on the storage account) — automatic transitions:
// {
//   "rules": [{
//     "name": "archive-old-files",
//     "definition": {
//       "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["documents/"] },
//       "actions": { "baseBlob": {
//         "tierToCool":    { "daysAfterModificationGreaterThan": 30 },   // 30d → cool
//         "tierToArchive": { "daysAfterModificationGreaterThan": 90 },   // 90d → archive
//         "delete":        { "daysAfterModificationGreaterThan": 2555 }  // 7y → delete
//       }}
//     }
//   }]
// }

Why it works: Access patterns are predictable: most files are accessed heavily when new and rarely after a while. Tiering aligns cost with that reality automatically — recent files stay on fast/hot storage, ageing files drift to progressively cheaper tiers, and truly expired data is deleted. The lifecycle policy does this without application code, based on age or last-access rules you define once.

Benchmark note: Storage tiering can cut storage costs dramatically for large, ageing datasets — archive tiers are often a fraction of hot-tier storage price, so moving years of rarely-touched documents to archive saves substantially at scale. The trade-off is retrieval: archive-tier data is cheap to store but slow (hours of rehydration latency) and more expensive to access, so it suits compliance archives and old backups, not anything that might be needed promptly. Model your access patterns before setting aggressive tiering — moving data to archive that then gets accessed frequently costs more than leaving it hot. The deletion rules are equally valuable: automatically purging data past its retention period controls both cost and compliance risk.

---

Choosing the right approach

• Receiving uploads correctly (baseline) → Streaming upload (Way 1)
• High-volume / large uploads, offload servers → Presigned direct-to-storage (Way 2)
• Serving files to many / distant users → CDN delivery (Way 3)
• Very large files on unreliable networks → Chunked / resumable upload (Way 4)
• Controlling cost of accumulating data → Tiered storage + lifecycle (Way 5)

The unifying principle: your application servers should orchestrate file operations, not be the pipe the bytes flow through. Keep large files out of application memory (stream, never buffer — Way 1), and ideally off your servers entirely (direct-to-storage uploads — Way 2; CDN downloads — Way 3) so your instances stay small and your capacity is spent on logic, not byte-shuffling. Layer on resumability (Way 4) when files are large and networks unreliable, and tiering (Way 5) to keep storage costs aligned with access reality. And never store large files in the database — it bloats backups, slows every query, and wastes the most expensive storage you have on what object storage does better and cheaper; keep a reference (the blob name/URL) in the database and the bytes in blob storage. The recurring theme echoes the large-data and caching use cases: move bulk bytes through infrastructure built for it, and keep your application on the control path, not the data path.

Note: this is an informational engineering overview. Cost and performance characteristics vary by cloud provider, region, and configuration. Always model your own access patterns and validate against your provider's current pricing and limits.