Explanation of NetCore 3.0 File Upload and Large File Upload Restrictions

Two ways to upload NetCore files

The two official file uploading methods given by NetCore are "buffering" and "streaming". Let me briefly talk about the difference between the two,

1. Buffer: First save the whole file to memory through model binding, and then we get stream through IFormFile, which has the advantage of high efficiency and large memory requirement. The file should not be too large.

2. Streaming processing: directly read the stream corresponding to the Section loaded by the request body and directly operate strem. You don't need to read the entire request body into memory,

The following is the official Microsoft statement

Buffer

The entire file is read into IFormFile, which is the C # representation of the file and is used to process or save the file. The resources (disk, memory) used for file uploads depend on the number and size of concurrent file uploads. If the application tries to buffer too many uploads, the site will crash when there is not enough memory or disk space. If the size or frequency of file uploads consumes application resources, use streaming.

Streaming processing

Receive a file from a multi-part request, and then apply to process or save it directly. Streaming does not significantly improve performance. Streaming reduces memory or disk space requirements when uploading files.

File size limit

Speaking of size limits, we have to start from two aspects, 1 application server Kestrel 2. application program (our netcore program),

1. Application Server Kestre Settings

The application server Kestrel limits us mainly to the size of the entire request body, which can be set by the following configuration (Program- > CreateHostBuilder), if it exceeds the setting range, it will be reported BadHttpRequestException: Request body too large Exception information

public static IHostBuilder CreateHostBuilder(string[] args) =>
 Host.CreateDefaultBuilder(args)
 .ConfigureWebHostDefaults(webBuilder =>
 {
  webBuilder.ConfigureKestrel((context, options) =>
  {
  // Setting up the application server Kestrel Maximum size of request body 50MB
  options.Limits.MaxRequestBodySize = 52428800;
  });
  webBuilder.UseStartup<Startup>();
});

2. Application settings

Application Settings (Startup- > InvalidDataException) Exception message will be reported if it exceeds the setting range

services.Configure<FormOptions>(options =>
 {
  options.MultipartBodyLengthLimit = long.MaxValue;
 });

Reset the size limit of file upload by setting.

Source code analysis

Here I'm talking about the parameter MultipartBodyLengthLimit under 1, which mainly limits the length of each file when we upload it in a "buffered" form. Why is it in the buffer form, because the help class we use to read the uploaded file in the buffer form is Read method under MultipartReaderStream class. This method will update the total number of byte read under each read once, and when it exceeds this number, it will be thrown throw new InvalidDataException($"Multipart body length limit {LengthLimit.GetValueOrDefault()} exceeded."); It is mainly reflected in the judgment of _ observedLength by UpdatePosition method

The following is the source code of two methods of MultipartReaderStream class. For convenience of reading, I have simplified some codes

Read

public override int Read(byte[] buffer, int offset, int count)
 {
  
  var bufferedData = _innerStream.BufferedData;
  　　　　 　　 int read;
 　　　　  read = _innerStream.Read(buffer, offset, Math.Min(count, bufferedData.Count));
  return UpdatePosition(read);
}

UpdatePosition

private int UpdatePosition(int read)
 {
  _position += read;
  if (_observedLength < _position)
  {
  _observedLength = _position;
  if (LengthLimit.HasValue && _observedLength > LengthLimit.GetValueOrDefault())
  {
   throw new InvalidDataException($"Multipart body length limit {LengthLimit.GetValueOrDefault()} exceeded.");
  }
  }
  return read;
}

From the code, we can see that when you do the restriction of MultipartBodyLengthLimit, the total amount of reading will be accumulated after each reading, and when the total amount of reading exceeds

The MultipartBodyLengthLimit setting throws an InvalidDataException exception,

Eventually upload my file to Controller as follows

It should be noted that BodyLengthLimit was not set when we created MultipartReader (this parameter will be passed to MultipartReaderStream.LengthLimit ), which is our ultimate limit, there is no limit if I don't set the value here, which can be reflected by the UpdatePosition method

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.WebUtilities;
using Microsoft.Net.Http.Headers;
using System.IO;
using System.Threading.Tasks;
 
namespace BigFilesUpload.Controllers
{
 [Route("api/[controller]")]
 public class FileController : Controller
 {
 private readonly string _targetFilePath = "C:\\files\\TempDir";
 
 /// <summary>
 ///  Streaming file upload 
 /// </summary>
 /// <returns></returns>
 [HttpPost("UploadingStream")]
 public async Task<IActionResult> UploadingStream()
 {
 
  // Get boundary
  var boundary = HeaderUtilities.RemoveQuotes(MediaTypeHeaderValue.Parse(Request.ContentType).Boundary).Value;
  // Get reader
  var reader = new MultipartReader(boundary, HttpContext.Request.Body);
  //{ BodyLengthLimit = 2000 };//
  var section = await reader.ReadNextSectionAsync();
 
  // Read section
  while (section != null)
  {
  var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition);
  if (hasContentDispositionHeader)
  {
   var trustedFileNameForFileStorage = Path.GetRandomFileName();
   await WriteFileAsync(section.Body, Path.Combine(_targetFilePath, trustedFileNameForFileStorage));
  }
  section = await reader.ReadNextSectionAsync();
  }
  return Created(nameof(FileController), null);
 }
 
 /// <summary>
 ///  Cached file upload 
 /// </summary>
 /// <param name=""></param>
 /// <returns></returns>
 [HttpPost("UploadingFormFile")]
 public async Task<IActionResult> UploadingFormFile(IFormFile file)
 {
  using (var stream = file.OpenReadStream())
  {
  var trustedFileNameForFileStorage = Path.GetRandomFileName();
  await WriteFileAsync(stream, Path.Combine(_targetFilePath, trustedFileNameForFileStorage));
  }
  return Created(nameof(FileController), null);
 }
 
 
 /// <summary>
 ///  Write files to disk 
 /// </summary>
 /// <param name="stream"> Stream </param>
 /// <param name="path"> File save path </param>
 /// <returns></returns>
 public static async Task<int> WriteFileAsync(System.IO.Stream stream, string path)
 {
  const int FILE_WRITE_SIZE = 84975;// Write out buffer size 
  int writeCount = 0;
  using (FileStream fileStream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.Write, FILE_WRITE_SIZE, true))
  {
  byte[] byteArr = new byte[FILE_WRITE_SIZE];
  int readCount = 0;
  while ((readCount = await stream.ReadAsync(byteArr, 0, byteArr.Length)) > 0)
  {
   await fileStream.WriteAsync(byteArr, 0, readCount);
   writeCount += readCount;
  }
  }
  return writeCount;
 }
 
 }
}

Summary:

It is also important to pay attention if you deploy on iis or other application servers such as Nginx, because they also have restrictions on the request body. It is also worth noting that the buffer size should not exceed the limit of netcore large objects when creating file stream objects. In this way, it is easy to trigger the recovery of the 2nd generation GC when the concurrency is high.