Nginx uses Gzip algorithm to compress messages in detail

What is HTTP compression

Sometimes large message data will be transmitted between the client and the server, which takes up a large network bandwidth and time. In order to save bandwidth and speed up the response speed of the message, the transmitted message data can be compressed first and then transmitted.

HTTP supports a variety of message compression algorithms. The following is a common request header. From Accept-Encoding fields, it can be seen that gzip, deflate and br compression algorithms are supported. In this paper, we focus on using Gzip algorithm to compress messages, such as Gzip to compress HTML, Javascript and CSS files. After compression, it can greatly reduce the amount of data transmitted by the network and improve the speed of users displaying web pages.

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Host: localhost:8000
If-Modified-Since: Tue, 21 Apr 2020 14:09:01 GMT
If-None-Match: "5e9efe7d-264"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Any technology is two-sided. Although HTTP compression can reduce bandwidth occupation and speed up response, it will occupy some computing resources on the client or server side because of the extra compression and decompression process.

Readers of HTTP know that the newspaper style of HTTP can be encoded and encrypted. In fact, HTTP compression is a special coding method, which can greatly reduce the number of messages and restore the original messages by using the corresponding solution method. (We can see that the essence of compression technology is actually a coding method.)

Use Scenarios of HTTP Compression

As can be seen from the above introduction of HTTP compression, this technology is an optimization technology, which is often used to compress the messages returned by the server to save bandwidth and speed up response.

The following is a brief introduction to the process of using Gzip compression for an HTTP.

The browser sends Http request to the Web server, and Accept-Encoding is included in request: gzip, deflate and br. (Tells the server that the browser supports gzip compression) After the Web server receives the request, it becomes the original Response, including the original Content-Type and Content-Length. The Web server encodes the Response through the Gzip. The encoded header includes Content-Type and Content-Length (compressed size), and adds Content-Encoding: gzip. Then the Response is sent to the browser. After the browser receives Response, it decodes Response according to Content-Encoding: gzip. After getting the original response, the web page is displayed.

The client can also send compressed data to the server, and decompress the requested data by code. For specification, Content-Encoding: gzip should also be added to the request

Realization of HTTP compression with Nginx

Nginx provides support for HTTP Gzip compression. Here, let's look at how to compress the returned message according to Nginx.

Gzip functions are supported in Nginx by ngx_http_gzip_module module, ngx_http_gzip_static_module module and ngx_http_gunzip_module module. 1 In general, Nginx compiles by default
These modules, you can use the nginx-V command to see if you install nginx contains these modules.

Gzip-related instructions may be in an http block, an server block, or an location block of the configuration file.

ngx_http_gzip_module module

ngx_http_gzip_module module is mainly responsible for opening and setting Gzip function, and compressing response data online and in real time. The module contains the following main instructions.

# 开启或者关闭Gzip功能,默认情况下，该指令设置为off，即不启用Gzip功能。只有将该指令设置为on时，其他指令设置才有效
gzip on | off

# 设置Gzip压缩文件使用缓存空间的大小
# 默认值是：gzip_buffers 32 4k|16 8k
gzip_buffers number size;

# 该指令用于设定Gzip压缩程度，包括级别1到级别9。
# 级别1表示压缩程度最低，压缩效率最高；级别9表示压缩程度最高，压缩效率最低，最费时间。
# 默认是1
gzip_comp_level level

# 针对不同种类客户端发起的请求，可以选择性地开启和关闭Gzip功能。
# 支持正则表达式，其中，regex 根据客户端的浏览器标志（User-Agent，UA）进行设置。
gzip_disable regex ...;

# 该设置使用了正则表达式，其可以匹配UC字符串中包含MSIE 4、MSIE 5和MSIE6的所有浏览器。
# 响应这些浏览器发出的请求时，Nginx服务器不进行Gzip压缩。
gzip_disable MSIE [4-6]\.;

# 早期的1些浏览器或者HTTP客户端，可能不支持Gzip自解压，因此用户有时会看到乱码，所以针
# 对不同的HTTP协议版本，需要选择性地开启或者关闭Gzip功能。该指令用于设置开启Gzip功能的最低HTTP协议版本。
# 默认设置为1.1版本，即只有客户端使用1.1及以上版本的HTTP协议时，才使用Gzip功能对响应输出数据进行压缩。
# 从目前来看，绝大多数的浏览器都支持Gzip自解压，1般采用默认值即可.
zip_http_version 1.0 | 1.1;

# 该指令设置页面的字节数，当响应页面的大小大于该值时，才启用Gzip功能。
# 建议设置成gzip_min_length 1024;
gzip_min_length length;

# 用于设置Nginx服务器是否对后端服务器返回的结果进行Gzip压缩;
# 1般情况下，后端都是用来做restAPI接口，返回的数据量不会太大，不建议进行压缩
# 真的需要对后端返回的数据进行压缩是可以再看下这块的内容
gzip_proxied off | expired | no-cache | no-store | private | no_last_modified | no_etag | auth | any ...;

# 设置MIME类型，被设置的类型将被压缩，默认值是：text/html
# 该变量还可以取“*”，表示对所有MIME类型的页面数据进行Gzip压缩
# 1般可以设置成：gzip_types text/plain application/javascript text/css text/xml
gzip_types mime-type ...;

# 开启后的效果是在响应头部添加了Accept-Encoding: gzip
gzip_vary on | off;

ngx_http_gzip_static_module Module

The ngx_http_gzip_static_module module is primarily responsible for searching and sending data pre-compressed by the Gzip function. This data is stored on the server with ". gz" as the suffix. If the data requested by the client has been compressed before and the client browser supports Gzip compression, the compressed data is returned directly.

The main difference between this module and ngx_http_gzip_module module is that this module uses static compression, and the HTTP response header contains Content-Length header field to indicate the length of the newspaper style, which can be used by the server to determine the length of response data; The latter uses Chunked encoded dynamic compression by default, which is mainly suitable for the situation that the server cannot determine the response data length, such as the situation of downloading large files, when the data length needs to be generated in real time.

The use of this module instruction is similar to that of the ngx_http_gzip_static_module module, so it will not be expanded here. You can refer to the official documents

This module is an optional HTTP module for the Nginx server and, if used, must be added with the with-http_gzip_static_module instruction during Nginx program configuration.

ngx_http_gunzip_module Module

The Nginx server supports Gzip compression of the response output data stream, which requires the client browser to have the ability to decompress and process Gzip compressed data, but if the client itself does not support this function, it requires the Nginx server to decompress the data before sending it. These compressed data may come from back-end server compression generation or Nginx server pre-compression generation. The ngx_http_gunzip_module module is used to decompress the compressed data for client browsers that do not support Gzip compressed data processing.

Similarly, the use of instructions for this module will not be specifically expanded, and you can refer to the official documents

Modern browsers 1 generally support compression, so this module is less likely to be used.

Configure column

gzip   on;
gzip_min_length 1000;
gzip_proxied expired no-cache no-store private auth;
gzip_types  text/plain application/xml;

To enable the Nginx server to apply the Gzip compression function globally, the Gzip configuration can be placed in the http global block. If we want to treat each virtual host differently, we can use the corresponding server
Add respective Gzip configuration instructions to the block;

Reading extension

Usually, not all applications developed may use Nginx. Let's see how other Web servers turn on support for HTTP compression.

1. Tomcat embedded in Spring Boot turns on compression function

Tomcat, as servet container + http server, also supports gzip compression. With the traditional Tomcat, we only need to turn on HTTP compression in server. xml configuration.
Under embed version, it needs to be configured by code. spring-boot The built-in tomcat is the embed version, and some default tomcat configurations have been made through the built-in autoconfig mechanism, but for some infrequent/advanced configurations, spring-boot does not provide an entry.

However, due to the features of spring bean, the default assembly of bean can be overridden, including tomcat-related configurations. The compression configuration can be turned on using the TomcatConnectorCustomizer interface.

public class ConnC1 implements TomcatConnectorCustomizer{

 @Override
 public void customize(Connector connector) {
  ProtocolHandler protocolHandler = connector.getProtocolHandler();
  if(protocolHandler instanceof Http11NioProtocol){
   Http11NioProtocol http11NioProtocol = (Http11NioProtocol)protocolHandler;
   http11NioProtocol.setCompression("on");//default off
   http11NioProtocol.setCompressibleMimeType();
   http11NioProtocol.setCompressionMinSize(2048);//default 2048(B)
   http11NioProtocol.setMaxKeepAliveRequests(1);//default 200
  }
 }
}

For clues about Tomcat's support for HTTP compression, you can start with the CompressionConfig class of Tomcat.

In fact, if you simply turn on the support for compression function, you only need to make the following configuration in Spring Boot:

server:
 compression:
 enabled: true
 min-response-size: 1024
 mime-types:
  application/json

Summarize