Of Stack

nginx agent repeatedly 302 solution of nginx Follow 302

  • 2020-05-15 03:41:59
  • OfStack

Use proxy_intercept_errors and recursive_error_pages agents 302 times

302 is a frequently used status code in the HTTP protocol. It is one of many redirection methods, and its semantics are often interpreted as "Moved Temporarily". By the way, the 302 used in practice is often misused (mixed with 303,307), and in HTTP/1.1 it means "Found".

Sometimes it's obvious, sometimes it's hidden. In the simplest case, when we enter a url A into the browser, the url bar will automatically jump to B and open a page, which is likely to be 302.

A more subtle situation often occurs when a player is embedded in a web page. For example, when you open a youku video playing page, grab the bag and observe 1, you will often find the shadow of 302. However, since these url are not opened directly in the browser, no changes can be seen in the browser's address bar. Of course, if the specific url is deliberately selected and copied into the browser's address bar, you can still observe it.

Youku was mentioned in the last paragraph. In fact, most online video sites now will use 302, the reason is very simple, video site traffic 1 as large, will use CDN, only in the difference between self-built CDN or commercial CDN. And because of the redirection semantics of 302 (again, the semantics of 302 are widely misused, when using 302, we should probably use 303 or 307, but we won't get into that point later), it can be well combined with the scheduling in CDN.

Let's look at an example. Open a netease video playing page, grab 1 of the package, and find the url in state 302. Such as:

[

http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

]

If we copy it into the browser address bar, we will find that the address bar quickly changes to another url. This Url is not fixed, and it may be:

[

http://14.18.140.83/f6c00af500000000-1408987545-236096587/data6/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

]

Use the curl tool to see the process more clearly: 


curl -I "http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L
HTTP/1.1 302 Moved Temporarily 
Server: nginx 
Date: Mon, 25 Aug 2014 14:49:43 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
NG: CCN-SW-1-5L2 
X-Mod-Name: GSLB/3.1.0 
Location: http://119.134.254.9/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 

HTTP/1.1 302 Moved Temporarily 
Server: nginx 
Date: Mon, 25 Aug 2014 14:49:41 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
X-Mod-Name: Mvod-Server/4.3.3 
Location: http://119.134.254.7/cc89fdac00000000-1408983581-2095617481/data4/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 
NG: CHN-SW-1-3Y1 

HTTP/1.1 200 OK 
Server: nginx 
Date: Mon, 25 Aug 2014 14:49:41 GMT 
Content-Type: video/mp4 
Content-Length: 3706468 
Last-Modified: Mon, 25 Aug 2014 00:23:50 GMT 
Connection: keep-alive 
Cache-Control: no-cache 
ETag: "53fa8216-388e64" 
NG: CHN-SW-1-3g6 
X-Mod-Name: Mvod-Server/4.3.3 
Accept-Ranges: bytes

As you can see, there are two 302's in between.

To put this example aside for a moment, let's talk about another important term: proxy. We often joke that some leaders are of type 302 and some are of type proxy. A 302-type leader will quickly transfer a task to others once it has passed through his hands, while an proxy leader will get involved in the task and even finish it all.

Going back to the above example, if there are multiple 302s in the middle of a visit to an url, what if you need to design an proxy with Nginx to hide all the 302s in the middle?

1. The original Proxy

As we know, Nginx itself is an excellent proxy server. So, first let's set up an Nginx forward proxy with server IP at 192.168.109.128 (one of my test virtual machines).

The initial configuration is simplified as follows: 


server {
    listen 80;
    location / {
        rewrite_by_lua '
            ngx.exec("/proxy-to" .. ngx.var.request_uri)
        ';
    }

    location ~ /proxy-to/([^/]+)(.*) {
        proxy_pass http://$1$2$is_args$query_string;

    }
}

The implementation function is when used

[

http://192.168.109.128/xxxxxx

]

When the agent is accessed, proxy goes to the real server represented by xxxxxx.

The test results are as follows: 


curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L
HTTP/1.1 302 Moved Temporarily 
Server: nginx/1.4.6 
Date: Mon, 25 Aug 2014 14:50:54 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
NG: CCN-SW-1-5L2 
X-Mod-Name: GSLB/3.1.0 
Location: http://183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 

HTTP/1.1 302 Moved Temporarily 
Server: nginx 
Date: Mon, 25 Aug 2014 14:50:55 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
X-Mod-Name: Mvod-Server/4.3.3 
Location: http://183.61.140.20/540966e500000000-1408983655-236096587/data1/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 
NG: CHN-ZJ-4-3M4 

HTTP/1.1 200 OK 
Server: nginx 
Date: Mon, 25 Aug 2014 14:50:55 GMT 
Content-Type: video/mp4 
Content-Length: 3706468 
Last-Modified: Mon, 25 Aug 2014 00:31:03 GMT 
Connection: keep-alive 
Cache-Control: no-cache 
ETag: "53fa83c7-388e64" 
NG: CHN-ZJ-4-3M4 
X-Mod-Name: Mvod-Server/4.3.3 
Accept-Ranges: bytes

As you can see, although proxy is used, the process is no different from the original access. The access process is, when the access

[

http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

]

When, Nginx will send the request proxy to

[

http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

]

While the latter will soon return to 1, 302, so Nginx as proxy, the 302 back to the client, the client to initiate the request, then repeat many times 302. Explain a problem here, 1 denier Nginx proxy backend back after 302, the client is, to break with the proxy Nginx Nginx cannot have the effect of full agent.

2. The first revision

Change the configuration file to: 


server {
    listen 80;
    location / {
        rewrite_by_lua '
            ngx.exec("/proxy-to" .. ngx.var.request_uri)
        ';
    }

    location ~ /proxy-to/([^/]+)(.*) {
        proxy_pass http://$1$2$is_args$query_string;
        error_page 302 = @error_page_302;

    }
    location @error_page_302 {
        rewrite_by_lua '
            local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")
            ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location
            ngx.exec("/proxy-to" .. upstream_http_location);
        ';

    }
}

The difference is that one error_page is used, so that when the backend of proxy is found to return 302, location for the purpose of this 302 is used to continue proxy instead of returning directly to the client. And this logic contains the idea of recursion, which is to trace 302 to the address of 200. The test results are as follows: 


curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L
HTTP/1.1 302 Moved Temporarily 
Server: nginx/1.4.6 
Date: Mon, 25 Aug 2014 15:01:17 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
NG: CCN-SW-1-5L2 
X-Mod-Name: GSLB/3.1.0 
Location: http://183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 

HTTP/1.1 302 Moved Temporarily 
Server: nginx 
Date: Mon, 25 Aug 2014 15:01:17 GMT 
Content-Type: text/html 
Content-Length: 154 
Connection: keep-alive 
X-Mod-Name: Mvod-Server/4.3.3 
Location: http://183.61.140.20/a90a952900000000-1408984277-236096587/data1/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 
NG: CHN-ZJ-4-3M4 

HTTP/1.1 200 OK 
Server: nginx 
Date: Mon, 25 Aug 2014 15:01:17 GMT 
Content-Type: video/mp4 
Content-Length: 3706468 
Last-Modified: Mon, 25 Aug 2014 00:31:03 GMT 
Connection: keep-alive 
Cache-Control: no-cache 
ETag: "53fa83c7-388e64" 
NG: CHN-ZJ-4-3M4 
X-Mod-Name: Mvod-Server/4.3.3 
Accept-Ranges: bytes

Visible, this modification still did not succeed!

Why is that? In analysis 1, we have added a header print statement to the location @error_page_302, but in the test, the header was not typed out, so it can be seen that the process did not enter the location @error_page_302.

The reason is that 


error_page 302 = @error_page_302;

error_page is the default return code for this processing. As proxy, as long as the response of the upstream server is forwarded successfully in this process, the status code should be 200. That is to say, what we really need to check is the status code returned by the back-end server of proxy, rather than the status code returned by proxy itself. Check 1 for wiki,proxy_intercept_errors instructions that do just that: 


Syntax: proxy_intercept_errors on | off;
Default:  
proxy_intercept_errors off;
Context:  http, server, location
Determines whether proxied responses with codes greater than or equal to 300 should be passed to a client or be redirected to nginx for processing with the error_page directive.

3. Second revision 


server {
    listen 80;
    proxy_intercept_errors on;
    location / {
        rewrite_by_lua '
            ngx.exec("/proxy-to" .. ngx.var.request_uri)
        ';
    }
    location ~ /proxy-to/([^/]+)(.*) {
        proxy_pass http://$1$2$is_args$query_string;
        error_page 302 = @error_page_302;

    }
    location @error_page_302 {
        rewrite_by_lua '
            local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")
            ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location
            ngx.exec("/proxy-to" .. upstream_http_location);
        ';
    }
}

Compared with the previous modification, the only difference is the addition of one proxy_intercept_errors directive. The test results are as follows: 


curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L 
HTTP/1.1 302 Moved Temporarily
Server: nginx/1.4.6
Date: Mon, 25 Aug 2014 15:05:54 GMT
Content-Type: text/html
Content-Length: 160
Connection: keep-alive
zzzz: /proxy-to/183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

This time is even more amazing, return to 1 302 status directly and finish, also do not continue to jump.

The problem is that, although the first 302 request successfully entered @error_page_302, the subsequent error_page directive failed. That is, error_page only checks the status code returned by the first backend and does not continue to check the subsequent backend status code.

Check 1, at which point another instruction, recursive_error_pages, comes in handy.

4. Third revision 


server {
    listen 80;
    proxy_intercept_errors on;
    recursive_error_pages on;
    location / {
        rewrite_by_lua '
            ngx.exec("/proxy-to" .. ngx.var.request_uri)
        ';
    }
    location ~ /proxy-to/([^/]+)(.*) {
        proxy_pass http://$1$2$is_args$query_string;
        error_page 302 = @error_page_302;

    }
    location @error_page_302 {
        rewrite_by_lua '
            local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")
            ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location
            ngx.exec("/proxy-to" .. upstream_http_location);
        ';
    }
}

Only the recursive_error_pages on instruction has been added compared to the previous one. The test results are as follows: 


server {
    listen 80;
    location / {
        rewrite_by_lua '
            ngx.exec("/proxy-to" .. ngx.var.request_uri)
        ';
    }

    location ~ /proxy-to/([^/]+)(.*) {
        proxy_pass http://$1$2$is_args$query_string;

    }
}
0

As you can see, Nginx finally managed to return 200. At this point, Nginx really plays the role of one Proxy, hiding one request from multiple 302 links and returning only one final result to the client.

5. Summary

To sum up, proxy_pass, error_page, proxy_intercept_errors, recursive_error_pages can be used together to hide the jump details of a request from the client and directly return the final result with a status code of 200 for the user.

Related articles: