Solve Syn Flooding problem when MySQL database encounters Syn Flooding

Syn attack is the most common and easily used attack technique, Taking advantage of the defects of TCP protocol, Send a large number of fake TCP connection requests, Common fake IP sends a large number of SYN packets, and the attacked server responds to SYN+ACK, because the other party is a fake IP, which will never receive packets and will not respond, resulting in the attacked server keeping a large number of semi-connections in SYN_RECV state, and retrying the default response handshake packet five times, filling the TCP waiting connection queue, and exhausting resources, so that normal business requests cannot be connected.

Syn attacks are common in application servers, while database servers should be difficult to encounter similar attacks in intranet. However, sometimes, if the application program is incorrectly connected with the database, it will be considered as Syn attacks on the database side, and the connection establishment will be refused.

[Problem Description]

The database suddenly refuses to link, and the application reports an error. At the time when the problem occurs, the following error information can be seen in the operating system log of the database server, namely/var/log/messages:

kernel: possible SYN flooding on port 3306. Sending cookies.

[Problem Analysis]

At the point of problem, from the monitoring index of database, Threads Connected has increased. This is also obvious, because for the database, Syn Flooding means that the application program suddenly initiates the connection to the database, and the operating system can't handle it, so report Syn Flooding. From the performance index of the database, the number of connections will definitely have a sudden increase. The solution is to analyze how these sudden growth came from, cut peaks and fill valleys, and make the connection more stable.

[Solution]

On the database server side, make the following adjustments: This adjustment means: increase the buffer of TCP semi-connection, the default value is 2048, and we adjust it to 8192, so that the anti-burst pressure of the system increases by 1. The default of Tcp_syn_retires and Tcp_synack_retires is 5, that is, the server will send packets five times before terminating the retry. We adjust this parameter to 2. Retry only once, so that the wrong packets can go wrong as early as possible to reduce the number of cached connections.

echo 8192 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 2 > /proc/sys/net/ipv4/tcp_syn_retries
echo 2 > /proc/sys/net/ipv4/tcp_synack_retries

This parameter adjustment takes effect immediately without restarting. Of course, after the server restarts, these parameters will also fall back to their default values. After this adjustment, the compression resistance of the database has been strengthened, but the problem has not been completely solved.

We also make corresponding adjustments on the client:

In order to reduce the pressure on the number of connections in the database, we usually recommend that the connection pool be configured as follows:

testWhileIdle = "false". Connection string health is not detected when idle
minIdle= "0". Minimum number of free connections in the connection pool
maxAge= "30000". How many milliseconds a link can be recycled.
initialSize= "1". Minimum number of initial connections in the connection pool
timeBetweenEvictionRunsMillis= "5000". Run interval of recycle thread (milliseconds)

For the present scenario, we suggest that the parameter minIdle be raised from 0 to 5. Let the connection pool usually have 5 free connections, so that these 5 free connections will be used first when requesting the database. Achieve the function of peak shaving and valley filling. Of course, the side effect is that the number of connections to the database will increase at ordinary times. The specific adjustment to how much is appropriate needs to be combined with the actual database connection load. For. NET program, there are also corresponding connection pool parameters can be adjusted: minPoolSize can be appropriately modified this parameter, also adjusted to 5.

After this adjustment, basically most of the database Syn Flooding problems can be solved.

Of course, these are all means of tuning, which can only improve the system slightly. Improve the ability to resist pressure. The final analysis depends on where the connection pressure comes from. And why you need to suddenly establish a large number of connections to the database. For this kind of burst scenario, whether it is appropriate to use a database. The alternative is to use Redis plus 1 layer buffer in front. Avoid sudden connection requests to the database. This involves the transformation of the application.

Summarize