An nginx 504 Gateway Time out error troubleshooting and resolution record

Remember a puzzling site lost response troubleshooting. Previously, the website has been using nginx as the proxy back-end apache to run php to provide services. apache will often appear irregularly and infrequently without the service being unresponsive, and then nginx will appear "504 Gateway Time-out "
Looking at the error log, you don't see anything either, assuming it's bug for apache (which it isn't, as we'll see below).

Perhaps people are too old to do anything and are willing to stay put, use monitoring tools and restart apache every time they receive an alarm. Finally, one day, I got bored and decided to deal with php. I didn't need to use apache head office. In my anger, I used the source to install php-fpm and transferred to php-fpm to run php. Installing php is not a hassle, the source installation is smooth, and the only thing you need to do is set up the output php error log for the php worker worker process.

When everything is ready, replace the original proxy_pass with fastcgipass.

upstream apachephp  {
    server www.jb51.net:8080; #Apache1
}
....
proxy_pass  http://apachephp;

I'll replace it with

upstream php {
        server 127.0.0.1:9000;
}
...
fastcgi_pass php;

You can move php running on apache to php-fpm running on php.
You think you can rest easy, and it's fine to complete the migration, but if you don't analyze the root cause of the problem. Problems still come to me, and the next day nginx offers gateway timeout for 504. Now that apache is out of the picture, apache is out of the picture.

That should still be on nginx and php-fpm, check the error log of nginx, you can see

[error] 6695#0: *168438 upstream timed out (110: Connection timed out) while reading response header from upstream,
...
request: "GET /kd/open.php?company=chinapost&number=PA24977020344 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.jb51.net"

You basically rule out nginx, nginx is waiting for php to process "GET /kd/ open.php? company=chinapost&number=PA24977020344 HTTP/1.1" timeout exits.

Restart php-fpm immediately, the problem is gone, the website is available.

When I visited the page again, there was still no response, but it was normal to visit another page at the same time. After the page was refreshed several times, the whole website was bad gateway timeout.

The problem is narrowed down to this php script.

netstat -napo |grep "php5-fpm" | wc -l

Check that the php worker process has reached the upper limit of 10 in the configuration file, and it feels like everyone is stuck with the script open.php.

What does this script do? This script is to collect express information, which USES php_curl.

The PHP script forces an exit if the execution time exceeds the php.ini configuration item max_execution_time with no results.

Check php.ini for max_execution_time, with a value of 30.

The universal google comes in handy. After google, you get

The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. The maximum execution time of any script that occurs during system calls, stream operations, database operations, etc., using system() is not included when the script has been run.

If you don't set a timeout, php will wait for the result of the call.

Check the open.php source file, and sure enough, there is no timeout for curl.

Add the following two lines, refresh, and the problem is solved.

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); //timeout on connect
curl_setopt($ch, CURLOPT_TIMEOUT, 10); //timeout on response

Of course, in addition to this method, php-fpm also provides an argument that we can force to kill a process that has been fruitless for a long time, except that this parameter is not turned on by default.

Es187en-fpm configuration file can set a parameter request_terminate_timeout, request termination timeout, when the request is executed beyond this time will be kill.

It also takes the parameter request_slowlog_timeout to log slow requests.

You can use this code to run php from the command line

$real_execution_time_limit = 60; // The time limit 
if (pcntl_fork())
{
// some long time code which should be
// terminated after $real_execution_time_limit seconds passed if it's not
// finished by that time
}
else
{
sleep($real_execution_time_limit);
posix_kill(posix_getppid(), SIGKILL);
}