Implementation of response compression in ASP. NET Core

Introduction #

Response compression technology is a commonly used technology in Web development field at present. Under the condition of limited bandwidth resources, using compression technology is the first choice to improve bandwidth load. We are familiar with Web servers, such as IIS, Tomcat, Nginx, Apache, etc., which can use compression technology. Commonly used compression types include Brotli, Gzip, Deflate, which have obvious effects on CSS, JavaScript, HTML, XML and JSON, but there are also certain restrictions that may not be so good for pictures, because pictures themselves are compressed formats. Second, for files smaller than about 150-1000 bytes (depending on the content of the file and the efficiency of compression, the overhead of compressing small files may result in larger compressed files than uncompressed files. In ASP. NET Core, we can use response compression in a very simple way.

Usage #

Using response compression in ASP. NET Core is relatively simple. First, add services. AddResponseCompression injection response compression related settings in ConfigureServices, such as compression type used, compression level, compression target type, etc. Secondly, add app to Configure. UseResponseCompression intercept request to judge whether compression is needed. The general usage mode is as follows

public class Startup
{
  public void ConfigureServices(IServiceCollection services)
  {
    services.AddResponseCompression();
  }

  public void Configure(IApplicationBuilder app, IHostingEnvironment env)
  {
    app.UseResponseCompression();
  }
}

If you need to customize 1 configuration, you can also set compression correlation manually

public void ConfigureServices(IServiceCollection services)
{
  services.AddResponseCompression(options =>
  {
    // You can add a variety of compression types, and the program will automatically get the best way according to the level 
    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();
    // Add a custom compression policy 
    options.Providers.Add<MyCompressionProvider>();
    // For the specified MimeType To use compression strategies 
    options.MimeTypes = 
      ResponseCompressionDefaults.MimeTypes.Concat(
        new[] { "application/json" });
  });
  // Set the corresponding compression level for different compression types 
  services.Configure<GzipCompressionProviderOptions>(options => 
  {
    // Use the fastest way to compress, single 1 It must be the best way to compress 
    options.Level = CompressionLevel.Fastest;

    // No compression operation 
    //options.Level = CompressionLevel.NoCompression;

    // Even if it takes a long time, use the best compression method 
    //options.Level = CompressionLevel.Optimal;
  });
}

The general way response compression works is to add Accept-Encoding to Request Header when initiating an Http request: gzip or any other compression type you want, and you can pass multiple types. After receiving the request, the server obtains Accept-Encoding to judge whether this type of compression mode is supported. If it is supported, the compression output content is related and Content-Encoding is set to return from the current compression mode 1. After receiving the response, the client obtains Content-Encoding to judge whether the server adopts compression technology, and judges which compression type is used according to the corresponding value, and then uses the corresponding decompression algorithm to obtain the original data.

Source code exploration #

Through the above introduction, I believe everyone has a certain understanding of ResponseCompression, and then we know its general working principle by looking at the source code.

AddResponseCompression#

First, let's look at the injection-related code, which is hosted in the ResponseCompressionServicesExtensions extension class [click to view the source code]

public static class ResponseCompressionServicesExtensions
{
  public static IServiceCollection AddResponseCompression(this IServiceCollection services)
  {
    services.TryAddSingleton<IResponseCompressionProvider, ResponseCompressionProvider>();
    return services;
  }

  public static IServiceCollection AddResponseCompression(this IServiceCollection services, Action<ResponseCompressionOptions> configureOptions)
  {
    services.Configure(configureOptions);
    services.TryAddSingleton<IResponseCompressionProvider, ResponseCompressionProvider>();
    return services;
  }
}

The main thing is to inject ResponseCompressionProvider and ResponseCompressionOptions. First, let's look at ResponseCompressionOptions [click to view the source code]

public class ResponseCompressionOptions
{
  //  Set the type of compression required 
  public IEnumerable<string> MimeTypes { get; set; }

  //  Set the type that does not require compression 
  public IEnumerable<string> ExcludedMimeTypes { get; set; }

  //  Whether to turn it on https Support 
  public bool EnableForHttps { get; set; } = false;

  //  Compressed type collection 
  public CompressionProviderCollection Providers { get; } = new CompressionProviderCollection();
}

I won't introduce too much about this class, which is relatively simple. ResponseCompressionProvider is the core class that we provide response compression algorithm, and how to choose compression algorithm automatically is provided by it. There are many codes in this class, so we will not explain them one by one. The specific source code can be consulted by ourselves [click to view the source code]. First, we will look at the constructor of ResponseCompressionProvider

public ResponseCompressionProvider(IServiceProvider services, IOptions<ResponseCompressionOptions> options)
{
  var responseCompressionOptions = options.Value;
  _providers = responseCompressionOptions.Providers.ToArray();
  // If the compression type is not set, the default is adopted Br And Gzip Compression algorithm 
  if (_providers.Length == 0)
  {
    _providers = new ICompressionProvider[]
    {
      new CompressionProviderFactory(typeof(BrotliCompressionProvider)),
      new CompressionProviderFactory(typeof(GzipCompressionProvider)),
    };
  }
  // According to CompressionProviderFactory Create the corresponding compression algorithm Provider For example GzipCompressionProvider
  for (var i = 0; i < _providers.Length; i++)
  {
    var factory = _providers[i] as CompressionProviderFactory;
    if (factory != null)
    {
      _providers[i] = factory.CreateInstance(services);
    }
  }
  // Set the default compression target type to text/plain , text/css , text/html , application/javascript , application/xml
  //text/xml , application/json , text/json , application/was
  var mimeTypes = responseCompressionOptions.MimeTypes;
  if (mimeTypes == null || !mimeTypes.Any())
  {
    mimeTypes = ResponseCompressionDefaults.MimeTypes;
  }
  // Will default MimeType Put in HashSet
  _mimeTypes = new HashSet<string>(mimeTypes, StringComparer.OrdinalIgnoreCase);
  _excludedMimeTypes = new HashSet<string>(
    responseCompressionOptions.ExcludedMimeTypes ?? Enumerable.Empty<string>(),
    StringComparer.OrdinalIgnoreCase
  );
  _enableForHttps = responseCompressionOptions.EnableForHttps;
}

BrotliCompressionProvider, GzipCompressionProvider is specific to provide compression methods of the place, let's look at the more commonly used Gzip Provider general implementation [click to view source code]

public class GzipCompressionProvider : ICompressionProvider
{
  public GzipCompressionProvider(IOptions<GzipCompressionProviderOptions> options)
  {
    Options = options.Value;
  }

  private GzipCompressionProviderOptions Options { get; }

  //  Corresponding Encoding Name 
  public string EncodingName { get; } = "gzip";

  public bool SupportsFlush => true;

  //  The core code is this sentence   Converts the original output stream to a compressed GZipStream
  //  What we set up Level The compression level will determine the performance and quality of compression 
  public Stream CreateStream(Stream outputStream)
    => new GZipStream(outputStream, Options.Level, leaveOpen: true);
}

As for other related methods of ResponseCompressionProvider, when we explain UseResponseCompression middleware, we will look at the methods used concretely, because this class is the core class of response compression, which is said in advance now, and we may forget where middleware is used. Next, let's look at the general implementation of UseResponseCompression.

UseResponseCompression#

UseResponseCompression is also a specific extension method without parameters, and it is relatively simple, because the configuration and work are completed by the injection place, so we directly view the implementation in the middleware and find the middleware location ResponseCompressionMiddleware [click to view the source code]

public class ResponseCompressionMiddleware
{
  private readonly RequestDelegate _next;
  private readonly IResponseCompressionProvider _provider;

  public ResponseCompressionMiddleware(RequestDelegate next, IResponseCompressionProvider provider)
  {
    _next = next;
    _provider = provider;
  }

  public async Task Invoke(HttpContext context)
  {
    // Determine whether to include Accept-Encoding Header information, not including direct shouting 1 Sound " Lift off 1 A "
    if (!_provider.CheckRequestAcceptsCompression(context))
    {
      await _next(context);
      return;
    }
    // Get the original output Body
    var originalBodyFeature = context.Features.Get<IHttpResponseBodyFeature>();
    var originalCompressionFeature = context.Features.Get<IHttpsCompressionFeature>();
    // Initialization response compression Body
    var compressionBody = new ResponseCompressionBody(context, _provider, originalBodyFeature);
    // Set to compression Body
    context.Features.Set<IHttpResponseBodyFeature>(compressionBody);
    context.Features.Set<IHttpsCompressionFeature>(compressionBody);

    try
    {
      await _next(context);
      await compressionBody.FinishCompressionAsync();
    }
    finally
    {
      // Restore the original Body
      context.Features.Set(originalBodyFeature);
      context.Features.Set(originalCompressionFeature);
    }
  }
}

This middleware is very simple, that is, it initializes ResponseCompressionBody. You may be curious to see that there is no trigger to call any code related to compression, ResponseCompressionBody only calls FinishCompressionAsync is related to release, don't worry, let's look at the structure of ResponseCompressionBody class

internal class ResponseCompressionBody : Stream, IHttpResponseBodyFeature, IHttpsCompressionFeature
{
}

This class implements IHttpResponseBodyFeature, and the Response. Body we use is actually the HttpResponseBodyFeature. Stream property we get. In fact, the methods related to Response. WriteAsync that we use are all calling PipeWriter internally for write operation, and PipeWriter comes from HttpResponseBodyFeature. Writer attribute. It can be roughly summarized as the core of output-related operations is to operate IHttpResponseBodyFeature. If you are interested, you can consult the relevant source code of HttpResponse by yourself. Therefore, our ResponseCompressionBody actually rewrites the output operation related methods. That is to say, as long as you call Write related to Response or Body related, you are actually operating IHttpResponseBodyFeature in essence. Since we open the middleware related to response output, we will call the method related to ResponseCompressionBody, the implementation class of IHttpResponseBodyFeature, to complete the output. There is still a deviation from our conventional understanding. In general, we think that it is enough to operate on the output Stream, but the response compression middleware actually rewrites the output-related operations.

After learning about this, I believe everyone will not have much doubt. Because ResponseCompressionBody overrides the output-related operations, Codes are relatively more, not one by one paste out, we only look at the design to response compression core-related code, about ResponseCompressionBody source code-related details can be interested in their own reference [click to view source code], the essence of the output is actually calling Write method, we will look at the implementation of Write method under 1

public override void Write(byte[] buffer, int offset, int count)
{
  // This is the core method to have the output related to compression here 
  OnWrite();
  //_compressionStream Initialized in the OnWrite In the method 
  if (_compressionStream != null)
  {
    _compressionStream.Write(buffer, offset, count);
    if (_autoFlush)
    {
      _compressionStream.Flush();
    }
  }
  else
  {
    _innerStream.Write(buffer, offset, count);
  }
}

From the above code, we see that the OnWrite method is the core operation, and we directly look at the OnWrite method implementation

private void OnWrite()
{
  if (!_compressionChecked)
  {
    _compressionChecked = true;
    // Determining whether the logic related to performing compression is satisfied 
    if (_provider.ShouldCompressResponse(_context))
    {
      // Matching Vary Value corresponding to header information 
      var varyValues = _context.Response.Headers.GetCommaSeparatedValues(HeaderNames.Vary);
      var varyByAcceptEncoding = false;
      // Judge Vary Is the value of Accept-Encoding
      for (var i = 0; i < varyValues.Length; i++)
      {
        if (string.Equals(varyValues[i], HeaderNames.AcceptEncoding, StringComparison.OrdinalIgnoreCase))
        {
          varyByAcceptEncoding = true;
          break;
        }
      }
      if (!varyByAcceptEncoding)
      {
        _context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);
      }
      // Get the best ICompressionProvider That is, the best compression mode 
      var compressionProvider = ResolveCompressionProvider();
      if (compressionProvider != null)
      {
        // Set the selected compression algorithm and put it into Content-Encoding In the value of the head 
        // Clients can use the Content-Encoding The header information determines which compression algorithm the server adopts 
        _context.Response.Headers.Append(HeaderNames.ContentEncoding, compressionProvider.EncodingName);
        // When compressing, set the  Content-MD5  Delete the header because the body content has changed and the hash is no longer valid. 
        _context.Response.Headers.Remove(HeaderNames.ContentMD5); 
        // When compressing, set the  Content-Length  Delete the header because the body content changes when the response is compressed. 
        _context.Response.Headers.Remove(HeaderNames.ContentLength);
        // Returns the compressed correlation output stream 
        _compressionStream = compressionProvider.CreateStream(_innerStream);
      }
    }
  }
}

private ICompressionProvider ResolveCompressionProvider()
{
  if (!_providerCreated)
  {
    _providerCreated = true;
    // Call ResponseCompressionProvider Returns the most appropriate compression algorithm 
    _compressionProvider = _provider.GetCompressionProvider(_context);
  }
  return _compressionProvider;
}

From the above logic, we can see that before executing the compression-related logic, we need to judge whether the compression-related method ShouldCompressResponse is satisfied. This method is the method in ResponseCompressionProvider, so we will no longer paste the code here. It is the judgment logic. I directly sorted out that it is roughly a few cases under 1

If the request is Https, is the EnableForHttps property setting of ResponseCompressionOptions set to allow compression under Https Response. Head cannot contain Content-Range header information Response. Head cannot contain Content-Encoding header information before Response Response. Head must include Content-Type header information before Response. Head The returned MimeType cannot contain the configured type that does not require compression, that is, ExcludedMimeTypes of ResponseCompressionOptions The returned MimeType needs to contain the configured type to be compressed, that is, MimeTypes of ResponseCompressionOptions If the above two conditions are not satisfied, the returned MimeType containing */* can also perform response compression

Next, let's look at the GetCompressionProvider method of ResponseCompressionProvider to see how it determines which 1 compression type to return

public void ConfigureServices(IServiceCollection services)
{
  services.AddResponseCompression(options =>
  {
    // You can add a variety of compression types, and the program will automatically get the best way according to the level 
    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();
    // Add a custom compression policy 
    options.Providers.Add<MyCompressionProvider>();
    // For the specified MimeType To use compression strategies 
    options.MimeTypes = 
      ResponseCompressionDefaults.MimeTypes.Concat(
        new[] { "application/json" });
  });
  // Set the corresponding compression level for different compression types 
  services.Configure<GzipCompressionProviderOptions>(options => 
  {
    // Use the fastest way to compress, single 1 It must be the best way to compress 
    options.Level = CompressionLevel.Fastest;

    // No compression operation 
    //options.Level = CompressionLevel.NoCompression;

    // Even if it takes a long time, use the best compression method 
    //options.Level = CompressionLevel.Optimal;
  });
}

0

Through the above introduction, we can roughly understand the general working mode of response compression, and briefly summarize 1

First, set the compression-related algorithm type or MimeType of the compression target Secondly, we can set the compression level, which will determine the compression quality and compression performance Through the response compression middleware, we can get a compression algorithm with the highest priority for compression, which is mainly for multiple compression types. This compression algorithm has a definite relationship with the internal mechanism and the order of registering compression algorithms, and will eventually choose the return with the largest weight. ResponseCompressionBody, the core work class of response compression middleware, rewrites the output related methods to compress the response by implementing IHttpResponseBodyFeature, instead of calling the related methods manually, it replaces the default output mode. As long as the response compression is set and the request satisfies the response compression, the default of the place where there is call output is to execute the compression related methods in ResponseCompressionBody, instead of intercepting the specific output for unified processing. As for why this is done, I haven't understood the designer's real consideration yet.

Summary #

Before looking at the relevant code, I thought that the logic related to response compression would be very simple. After reading the source code, I knew that I thought it was too simple. Among them, the biggest difference with my own ideas is that in ResponseCompressionMiddleware middleware, I thought it was to intercept the output stream through Unified 1 to compress the operation, but I didn't expect to rewrite the overall output operation. Because when we used Asp. Net related framework before, we wrote Filter or HttpModule for processing, so there is a mindset. Maybe the designer of Asp. Net Core has a deeper understanding, or maybe my understanding is not thorough enough to understand what the benefits of doing so are. If you have a better understanding or the answer, please leave a message in the comment area.