In depth analysis of Tomcat unresponsive problems and solutions

Problem description

There were several tomcat in the production environment, but suddenly we found that all the requests could not be answered. Because our web server used nginx, we would reverse the request to tomcat, so we suspected that nginx had not received the request at first. However, when we checked the log, we found that a large number of 499 returned in nginx, indicating that the problem was still on tomcat.

Troubleshoot problems

The first thing that comes to my mind is whether CPU is running full. Although CPU did not report to the police, I still instinctively ordered top to look at the system load. I found that the system only had load of 0.x and memory consumption of cpu was normal.

Since there is no abnormality in CPU, there should be no problem with GC, but we checked GC log, and sure enough, GC is no problem

At this point, jstack had to be brought into play, and sure enough, after using jstack, it was found that many threads were in WAITING state

"http-nio-127.0.0.1-801-exec-498" daemon prio=10 tid=0x00002ada7c14f800 nid=0x16a6 waiting on condition [0x00002ada9c905000]

　　java.lang.Thread.State: WAITING (parking)

　　at sun.misc.Unsafe.park(Native Method)

　　- parking to wait for <0x00000007873e6990> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

　　at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

　　at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

　　at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:133)

　　at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:282)

　　at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)

　　at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:177)

　　at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:170)

　　at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:102)

　　at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:240)

　　at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:227)

　　at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)

　　at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

　　at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)

　　at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

　　at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

　　at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

　　at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)

　　at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:105)

　　at com.weimai.utils.HttpClientUtil.doGet(HttpClientUtil.java:87)

　　at com.weimai.utils.WeiBoUtil.checkUser(WeiBoUtil.java:214)

　　at com.weimai.web.UserInfoController.newWeiboLogin(UserInfoController.java:1223)

　　at sun.reflect.GeneratedMethodAccessor390.invoke(Unknown Source)

　　at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

　　at java.lang.reflect.Method.invoke(Method.java:606)

At this time, I realized that the problem should appear on the http connection, and immediately checked the connection status of port 801 with netstat. Indeed, I found that many requests were CLOSE_WAIT. Here is a brief explanation of the state of CLOSE_WAIT

Because if the server end actively breaks the current connection, then it will take a total of 4 packet for both sides to close the TCP connection

server - > FIN - > client

server < - ACK < - client

At this point, the server end is in the FIN_WAIT_2 state, while our program is in the CLOSE_WAIT state

server < - FIN < - client

When client sends FIN to server,client is in the LAST_ACK state.

server - > ACK - > client

server responds to ACK, then client's socket will actually be in CLOSED state

Our request is in the state of CLOSE_WAIT, not LAST_ACK, indicating that FIN has not been sent to server, so it is very simple to see how HttpClientUtil is handled. Indeed, it is found in the code of HttpClientUtil that abort has not been used for the abnormally closed http connection. The problem is solved after adding and improving try catch finally block.