Handling Large Data Payloads in Your API

Use Case: Handling Large Data Payloads in Your API

The everyday problem: A client sends a 2 GB JSON import, a user uploads a multi-gigabyte video, or a report endpoint must return millions of rows. The naive approach — letting the framework read the whole request body into memory before your code runs, or loading an entire table into a List<T> before returning it — makes the server act like a sponge: it absorbs the entire payload into RAM at once. Memory spikes by gigabytes, the garbage collector stalls, and under concurrent load the process crashes. The challenge is moving large data in and out of your API without ever holding the whole payload in memory — keeping memory flat whether the data is 10 MB or 10 GB.

In enterprise development, handling large payloads well is one of the clearest dividing lines between standard engineers and architectural experts. Five mechanisms cover the territory, depending on whether the data is structured JSON, binary file uploads, very large transfers over unreliable networks, cloud-native uploads, or data leaving your system. The unifying idea behind all of them: treat data as a conveyor belt (process one piece, discard it, pull the next) rather than a sponge (absorb everything, then act).

---

Way 1: Raw Stream Deserialization (the JSON pipe)

Best used for: Massive incoming structured JSON — bulk imports, large data feeds, and webhooks from data providers — where the body is a big array of objects you want to process item by item.

The concept: By default, declaring a body parameter like [FromBody] List<ProductDto> tells ASP.NET Core to read and parse the entire request body into that parameter before your method runs — so a 2 GB payload spikes RAM by 2 GB before a single line of your code executes. Instead, you remove the body parameter entirely (telling the framework "hands off, don't buffer this"), grab the raw network stream from HttpContext.Request.Body, and feed it into JsonSerializer.DeserializeAsyncEnumerable<T>, which inflates one object at a time off the wire.

// BEFORE — buffering: the whole 2GB payload is read into RAM before this runs
[HttpPost("buffer")]
public IActionResult UploadLargeFile([FromBody] List<ProductDto> massiveList)
{
    foreach (var product in massiveList) // RAM already spiked by 2GB before reaching here
        SaveToDatabase(product);
    return Ok();
}

// AFTER — streaming: no body parameter, read the raw network stream directly
[HttpPost("stream")]
public async Task<IActionResult> UploadLargeFile()
{
    // No [FromBody] parameter — ASP.NET Core does NOT buffer or parse automatically.
    var rawNetworkStream = HttpContext.Request.Body;

    // Pull objects off the network wire one at a time, asynchronously
    var productStream = JsonSerializer.DeserializeAsyncEnumerable<ProductDto>(rawNetworkStream);

    await foreach (var product in productStream)
    {
        // Only ONE product lives in RAM at a time — memory stays flat
        if (product is not null)
            await SaveToDatabaseAsync(product);
    }
    return Ok();
}

// CLIENT side — stream the file from disk too, so neither side spikes RAM
public async Task UploadMassiveJsonAsync(string filePath)
{
    // Open a read handle — does NOT load the file into memory
    using var fileStream = File.OpenRead(filePath);

    var streamContent = new StreamContent(fileStream);
    streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");

    // HttpClient reads chunks from disk and pushes them straight onto the network pipe
    var response = await _httpClient.PostAsync("https://localhost:7001/api/products/stream", streamContent);
    response.EnsureSuccessStatusCode();
}

Why it works: The data becomes a continuous pipeline: the client reads a few kilobytes from disk and flushes them onto the network; the server receives those bytes via Request.Body; DeserializeAsyncEnumerable turns them into objects and hands them to your loop; the garbage collector reclaims each processed object while the next bytes are still arriving. Nothing ever holds the whole payload. Removing the body parameter is the essential trick — a declared parameter forces the framework to buffer and parse everything up front, defeating the entire approach.

Benchmark note: Memory footprint stays flat — typically under ~100 KB — whether the payload is 1 MB or 10 GB, because objects are inflated, processed, and collected in a rolling fashion rather than all held at once. The buffered version, by contrast, uses memory proportional to the payload size times the number of concurrent uploads, which is what crashes servers under load. The trade-off is that you give up automatic model binding and validation on the body, so you validate each item yourself inside the loop. For large structured-JSON ingestion, this is the foundational technique.

---

Way 2: Multipart Streaming via MultipartReader

Best used for: Traditional web-form uploads that carry large binary files (videos, zip archives, images) alongside other form fields (text, metadata) — the classic multipart/form-data upload.

The concept: The standard IFormFile interface forces the framework to buffer file parts into server RAM or temp disk before your code sees them. Instead, you disable the automatic model binding with a custom attribute and use Microsoft.AspNetCore.WebUtilities.MultipartReader to read the HTTP multipart boundaries yourself as an unbuffered stream, copying each file section straight to its destination.

[HttpPost("upload-multipart")]
[DisableFormValueModelBinding] // custom attribute — stops the framework auto-buffering the form
public async Task<IActionResult> UploadMultipart()
{
    var boundary = MultipartRequestHelper.GetBoundary(Request.ContentType);
    var reader   = new MultipartReader(boundary, Request.Body);

    var section = await reader.ReadNextSectionAsync();
    while (section != null)
    {
        if (ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var disposition))
        {
            // Stream this section's body straight to storage — never fully in RAM
            using var targetStream = File.Create(Path.Combine("uploads", Path.GetRandomFileName()));
            await section.Body.CopyToAsync(targetStream);
        }
        section = await reader.ReadNextSectionAsync();
    }
    return Ok();
}

Why it works: Reading the multipart sections manually means each file flows through a small buffer directly to its destination, so a large upload never materialises whole in memory. Disabling the default form model binding is what prevents the framework from buffering the parts before you get to them — the same "hands off" principle as Way 1, applied to multipart form data.

Benchmark note: This keeps memory flat for binary uploads of any size, where IFormFile would buffer to RAM or temp disk first (fine for small files, ruinous for large ones under concurrency). The cost is manual handling of the multipart protocol — boundaries, content-disposition parsing, and validating each section yourself. Always cap the maximum request size and validate file type/size as you stream, since an unbounded upload endpoint is a denial-of-service vector. Use this whenever real form uploads include large files.

---

Way 3: Client-Driven Chunked / Resumable Uploads

Best used for: Very large files — from a few gigabytes up to hundreds of gigabytes (raw video logs, medical imaging, datasets) — especially over unstable or mobile networks where a single failed transfer is unacceptable.

The concept: The front-end splits the file into small fixed-size chunks (e.g. 10 MB) and uploads each as a separate HTTP request carrying an upload ID and a chunk index. The API appends each chunk to a tracking file on disk or cloud storage. If the connection drops, the client asks the API for the current byte offset and resumes from the next missing chunk instead of restarting.

// Each chunk is an independent, retryable request — a dropped connection loses only one chunk
[HttpPost("upload-chunk")]
public async Task<IActionResult> UploadChunk(
    [FromHeader(Name = "Upload-Id")] string uploadId,
    [FromHeader(Name = "Chunk-Index")] int chunkIndex,
    CancellationToken ct)
{
    var path = Path.Combine("uploads", "partials", $"{uploadId}.part");
    // Append this chunk's bytes to the tracking file at the right offset
    await using var target = new FileStream(path, FileMode.Append, FileAccess.Write);
    await Request.Body.CopyToAsync(target, ct);
    return Ok(new { received = chunkIndex });
}

// Client resumes by querying current size, then sending only the missing chunks:
[HttpGet("upload-status/{uploadId}")]
public IActionResult Status(string uploadId)
{
    var path = Path.Combine("uploads", "partials", $"{uploadId}.part");
    long offset = System.IO.File.Exists(path) ? new FileInfo(path).Length : 0;
    return Ok(new { bytesReceived = offset }); // client resumes from here
}
// Often implemented with the open TUS resumable-upload protocol rather than by hand.

Why it works: Because each chunk is an independent request, a network failure costs only the in-flight chunk (a few megabytes) rather than the whole transfer. The client tracks progress and resumes from the last successful offset — so a Wi-Fi drop at 90% of a 50 GB upload resumes from chunk 46, not from zero. This turns "upload a huge file over flaky mobile data" from near-impossible into reliable.

Benchmark note: For large files on unreliable networks, resumability is the difference between uploads that complete and ones that perpetually fail and restart. Chunking also enables parallel chunk upload for higher throughput and real progress reporting. The cost is added complexity on both client and server — tracking chunk state, assembling the final file, and cleaning up abandoned partial uploads (a lifecycle rule to delete stale partials). Standard protocols like TUS implement this correctly so you don't hand-roll it. Reserve it for files genuinely large enough that whole-file uploads fail; for small files on good connections it is unnecessary overhead.

---

Way 4: Presigned Cloud Storage URLs (the backend bypass)

Best used for: High-volume cloud-native SaaS on AWS S3 or Azure Blob Storage, where you want large files to never touch your API servers at all.

The concept: Instead of routing file bytes through your API, you issue the client a temporary, cryptographically signed URL granting permission to upload one specific object, and the client uploads directly to cloud storage. Your API only authorises the operation — it never handles the bytes.

// 1. Client asks for permission: "I want to upload video.mp4 (5GB)"
// 2. API validates permissions and issues a short-lived presigned URL — no bytes pass through
[HttpPost("uploads/presign")]
public IActionResult Presign(string fileName)
{
    var blobName   = $"{Guid.NewGuid()}/{SafeName(fileName)}";
    var blobClient = _containerClient.GetBlobClient(blobName);

    var sas = blobClient.GenerateSasUri(new BlobSasBuilder
    {
        BlobContainerName = _containerClient.Name,
        BlobName          = blobName,
        Resource          = "b",
        ExpiresOn         = DateTimeOffset.UtcNow.AddMinutes(15) // short-lived
    }.SetPermissions(BlobSasPermissions.Write)); // write to this ONE blob only

    return Ok(new { uploadUrl = sas.ToString(), blobName });
}

// 3. Client PUTs the file directly to Azure/S3 using that URL (bypasses your API).
// 4. Client then notifies your API the upload is complete, passing the blob location.
// AWS S3 equivalent: GetPreSignedUrl with the PUT verb and a short expiry.

Why it works: Your application servers are removed from the data path entirely — they spend a millisecond issuing a URL instead of minutes shuffling gigabytes. The cloud storage service, built for exactly this and effectively unlimited in bandwidth, handles the transfer. Your server capacity stays free for application logic, and upload throughput is bounded by the cloud provider (enormous) rather than your servers (finite).

Benchmark note: This is transformative for file-heavy applications: upload capacity becomes the provider's near-unlimited bandwidth, and your instances stay small and cheap because they never move file bytes. The trade-offs are coordination and validation — the client makes two round-trips (presign, then completion notice), and because the file bypassed your API you validate after upload (size/type from blob metadata, malware scan), often triggered by a storage event. Scope the presigned URL tightly: one blob, write-only, short expiry, so it cannot be abused. For anything beyond modest files at modest volume, this is the right model.

---

Way 5: Non-Allocating Database Streaming (IAsyncEnumerable)

Best used for: Data egress — exporting millions of rows from a database to a client as a CSV or JSON array — the outbound mirror of Way 1.

The concept: The naive export queries the whole table into a giant in-memory List<T>, formats it, and returns it — causing a devastating memory spike. Instead, combine EF Core's AsNoTracking() with AsAsyncEnumerable() and return IAsyncEnumerable<T> directly, so rows stream from the database, through your API, to the HTTP response one record at a time without ever being fully materialised.

[HttpGet("export-millions-of-records")]
public IAsyncEnumerable<AuditLog> ExportData()
{
    // Rows stream from the database straight out to the HTTP response, one at a time.
    // AsNoTracking() skips change-tracking overhead; AsAsyncEnumerable() avoids buffering.
    return _dbContext.AuditLogs
        .AsNoTracking()
        .AsAsyncEnumerable();
    // No List<T>, no full materialisation — memory stays flat across millions of rows.
}

Why it works: Returning an async stream means the framework serialises each row to the response as it arrives from the database, then discards it — so application memory stays flat regardless of result-set size. AsNoTracking() removes the change-tracking overhead that would otherwise accumulate per entity, and the database, the API, and the client all work in lockstep on a rolling window of rows rather than the whole set.

Benchmark note: This keeps export memory flat whether you return a thousand rows or ten million, where the buffered "query into a List, then return" approach spikes memory proportionally to row count and routinely crashes on large exports. The considerations: keep the database connection open for the duration of the stream (which is the point, but means long-lived connections for big exports), and stream the output format incrementally too (write CSV rows or a JSON array element-by-element rather than building the whole string). This is the egress counterpart to Way 1 — same conveyor-belt principle, applied to data leaving your system.

---

Choosing the right approach

• Large incoming structured JSON (imports, feeds, webhooks) → Raw stream deserialization (Way 1)
• Web-form uploads with large binary files + fields → Multipart streaming via MultipartReader (Way 2)
• Very large files over unreliable networks → Client-driven chunked / resumable uploads (Way 3)
• Cloud-native, keep files off your servers entirely → Presigned cloud storage URLs (Way 4)
• Exporting millions of rows out of the database → Non-allocating database streaming (Way 5)

The mental model: the dividing line between code that crashes under large payloads and code that doesn't is sponge versus conveyor belt — never absorb the whole payload, always process it as a flowing stream. For data coming in, that means refusing the framework's default buffering: drop the body parameter and read Request.Body for JSON (Way 1), use MultipartReader for form uploads (Way 2), and chunk very large transfers so failures are cheap and resumable (Way 3). Better still for cloud systems, keep the bytes off your API entirely with presigned URLs (Way 4). For data going out, stream from the database with IAsyncEnumerable rather than materialising a giant list (Way 5). In every case memory stays flat regardless of payload size, because objects are created, processed, and garbage-collected in a rolling window while the next bytes are still in transit. This is precisely the architectural instinct that separates engineers who can build systems that survive real-world data volumes from those whose servers fall over the first time someone uploads a 2 GB file.

Note: this is an informational engineering overview. Memory figures are illustrative order-of-magnitude estimates that depend on payload shape, serialization, buffering, and infrastructure. Always set request-size limits, validate untrusted input, and measure against your own workload.