Limitations on the maximum number of connections for high concurrency socket under Linux

1, modify the user process can open the file number limit

On Linux platform, write a client or server application, in the high concurrency TCP connection processing, the highest number of concurrent to be system for single user 1 process at the same time to open the file number limitation (this is because the system for each TCP connection to create a socket handle, each socket handle is also a file handle). You can use the ulimit command to view the number of files the system allows the current user's process to open:

[speng@as4 ~]$ ulimit -n
1024

This means that the current user's each process up to allow open 1024 files at the same time, the 1024 file also need to remove each process must open the standard input, standard output and standard error, the server monitor socket, socket unix domain of communication between processes and other documents, then the rest of the number of files can be used for the client socket connection is only about 1024-10 = 1014 or so. This means that by default, Linux-based communication applications allow a maximum of 1014 simultaneous TCP connections.

For communication handlers that want to support a higher number of simultaneous connections to TCP, you must modify the soft (soft limit) and hard (hardlimit) limits that Linux has on the number of files open simultaneously by the current user's process. The soft limit refers to the limit of the number of files that Linux can open at the same time within the range of the current system. The hard limit is the maximum number of files the system can open at the same time calculated based on the system's hardware resource status (mainly system memory). Usually the soft limit is less than or equal to the hard limit.

The easiest way to modify the above restriction is to use the ulimit command:

[speng@as4 ~]$ ulimit -n

In the above command, specify the maximum number of files that the single 1 process is allowed to open to be set. If the system echo is something like "Operation notpermitted," then the above restriction failed, in fact because the value specified in "exceeds the soft or hard limit of the Linux system on the number of open files for that user. Therefore, it is necessary to modify the soft and hard limits that the Linux system imposes on users regarding the number of open files.

Step 1, modify/etc/security/limits conf file, add the following line in the file:

speng soft nofile 10240
speng hard nofile 10240

Where speng specifies which user's open file limit to modify, the '*' sign can be used to indicate the limit to modify all users; soft or hard specifies whether to modify the soft or hard limit; 10240 specifies the new limit value you want to modify, which is the maximum number of open files (note that the soft limit value is less than or equal to the hard limit). Save the file after you have modified it.

Step 2, modify/etc/pam d/login file, add the following line in the file:

session required /lib/security/pam_limits.so

This is tell Linux after the user login system, should call pam_limits. so module to set up the system of the user can maximum limit of the number of the use of various resources (including user limits of the maximum number of files can be opened), and pam_limits. so module from/etc security/limits conf reads the configuration file to set the limit. Save the file after you have modified it.

Step 3, view the maximum open file limit for Linux at the system level, using the following command:

[speng@as4 ~]$ cat /proc/sys/fs/file-max
12158

This indicates that this Linux system allows a maximum of 12,158 files to be opened simultaneously (that is, including the total number of files opened by all users), which is the hard limit at the Linux system level. No user-level open file limit should exceed this number. In general, this system-level hard limit is the optimal maximum open file limit calculated at startup for Linux based on the system's hardware resources. If there is no special need, this limit should not be modified, unless you want to set a value beyond this limit for user-level open file limits. The way to modify this hard limit is to modify the /etc/ rc.local script by adding the following line to the script:

echo 22158 > /proc/sys/fs/file-max

This is to force Linux to set the hard limit for system-level open files to 22158 after startup. Save the file after you have modified it.

When you restart the system after completing the above steps, you can generally set the maximum number of files that the Linux system allows to be opened at the same time by a single process for a specified user to the specified value. If, after a reboot, you use the ulimit-n command to check that the limit on the number of files a user can open is still lower than the maximum set in the above steps, this may be because the ulimit-n command in the user login script /etc/profile has already limited the number of files a user can open at the same time. Since the maximum number of files that a user can open at the same time is modified through ulimit-n, the new modified value can only be less than or equal to the value set last time by ulimit-n, it is impossible to increase this limit with this command. So, if you have the above problems, only to open/etc profile script file, the file to check whether or not to use the ulimit - n limits the user can open at the same time the maximum number of files, if found, then delete the line command, or it is set to change the value of the appropriate value, and then save the file, the user exit and log back in the system.

By following these steps, the system limits on the number of open files can be lifted for a communication handler that supports highly concurrent TCP connection processing.

2. Modify the network kernel's restrictions on TCP connection (refer to "optimizing kernel parameters" in the next article)

When writing a client communication handler on Linux that supports highly concurrent TCP connections, it is sometimes found that even though the system limit on the number of files users can open at the same time has been lifted, a new TCP connection cannot be successfully established when the number of concurrent TCP connections increases to a fixed number. There are several reasons for this.

The first reason may be that the Linux network kernel has a limited range of local port Numbers. At this point, the next step analyzes why the TCP connection cannot be established, and the problem is found in the failure of the return of the connect() call, and the view system error message is "Can't assign requestedaddress". Also, if you monitor the network at this point with the tcpdump tool, you will find that the client sends the network traffic of the SYN packet when there is no TCP connection at all. These facts suggest that the problem is that there are limitations in the local Linux system kernel. In fact, the root cause of the problem is that the TCP/IP protocol implementation module of the Linux kernel limits the range of local port Numbers that all clients in the system can connect to TCP (for example, the kernel limits the range of local port Numbers to between 1024 and 32768). When a 1 time in the system at the same time, there are too many TCP client connection, because every TCP client connection takes up 1 only 1 local port (the port number in the system of the local port range limit), if the existing TCP client connection has filled all the local port, is can't for the new TCP client connection assigned a local port, so the system will in this case in connect () call returns failure, Set the error message to "Can't assignrequested address". For these control logic, you can view the Linux kernel source code. Taking the linux2.6 kernel as an example, you can view the following functions in the tcp_ipv4.c file:

static int tcp_v4_hash_connect(struct sock *sk)

Note the access control to the variable sysctl_local_port_range in the above function. The initialization of the variable sysctl_local_port_range is set in the following function in the tcp.c file:

void __init tcp_init(void)

The local port number range that is set by default at kernel compile time may be too small, so you need to modify this local port range limit.

Step 1: modify the /etc/ sysctl.conf file by adding the following line to the file:

net.ipv4.ip_local_port_range = 1024 65000

This indicates that the system limits the local port range to between 1024 and 65000. Note that the minimum value of the local port range must be greater than or equal to 1024; The maximum port range should be less than or equal to 65535. Save the file after you have modified it.

Step 2, execute the sysctl command:

[speng@as4 ~]$ sysctl -p
If there is no error, the new local port range setting is successful. If the above port range is set, it is theoretically possible for a single process to establish up to 60,000 TCP client connections simultaneously.

The second reason for the failure to establish an TCP connection may be that the IP_TABLE firewall of the Linux network kernel has a limit on the maximum number of TCP connections to be traced. If you monitor the network with the tcpdump tool, you will find that the client sends the SYN packet network traffic when there is no TCP connection at all. Since IP_TABLE firewall in the kernel to every TCP connection status tracking, tracking information will be in the conntrackdatabase in the kernel memory, the size of the database is limited, when there is too much TCP connection system, database capacity is insufficient, IP_TABLE establish tracking information, can not connect to new TCP so as connect obstruction () call. At this point, the kernel limit on the maximum number of TCP connections to be traced must be modified in the same way that the kernel limit on the range of local port Numbers must be modified:

Step 1: modify the /etc/ sysctl.conf file by adding the following line to the file:

net.ipv4.ip_conntrack_max = 10240

This indicates that the system limit for the maximum number of TCP connections tracked is set to 10240. Note that this limit is kept as small as possible to save kernel memory usage.

Step 2, execute sysctl:
[speng@as4 ~]$ sysctl -p

If there is no error, the system has successfully modified the new maximum traced TCP connection limit. If the above parameters are set, a single process can theoretically establish up to 10,000 TCP client connections simultaneously.

3. Use programming techniques that support highly concurrent networks I/O

When writing highly concurrent TCP connected applications on Linux, you must use the appropriate network I/O technology and I/O event dispatch mechanism.

Available I/O technologies are synchronous I/O, non-blocking synchronous I/O(also known as reactive I/O), and asynchronous I/O. In the case of high TCP concurrency, if I/O is used synchronously, it will seriously block the operation of the program unless one thread is created for each I/O TCP connection. However, too many threads will cause huge overhead due to the scheduling of threads by the system. Therefore, it is not advisable to use synchronous I/O in the case of high TCP concurrency. Consider using non-blocking synchronous I/O or asynchronous I/O. Non-blocking synchronization techniques for I/O include the use of select(), poll(), epoll and other mechanisms. The asynchronous I/O technique USES AIO.

From the point of view of the I/O event dispatch mechanism, the use of select() is inappropriate because it supports a limited number of concurrent connections (typically no more than 1024). If performance is taken into account, poll() is also inappropriate. Although it can support a high TCP concurrency, it is quite inefficient when the concurrency is high due to its "polling" mechanism, and may have an uneven distribution of I/O events, resulting in "starvation" of I/O on some TCP connections. This is not the case with epoll or AIO. (the early AIO technical implementation of the Linux kernel was implemented by creating one thread per I/O request in the kernel. This implementation mechanism also had serious performance problems in the case of highly concurrent TCP connections. However, in the latest Linux kernel, the implementation of AIO has been improved).

To sum up, when developing Linux applications that support highly concurrent TCP connections, epoll or AIO technologies should be used to achieve I/O control on concurrent TCP connections. This will provide effective I/O assurance for the enhancement program to support highly concurrent TCP connections.

Optimization of kernel parameter sysctl.conf

conf /etc/ sysctl.conf is the configuration file used to control the linux network. It is very important for network-dependent applications (such as web server and cache server). RHEL provides the best tuning by default.

Recommended configuration (clear the original /etc/ sysctl.conf and copy the following) :

net.ipv4.ip_local_port_range = 1024 65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

This configuration refers to the recommended configuration for cache server varnish and the recommended configuration for SunOne server system optimization.

varnish tuning recommendations address is: http: / / varnish projects. linpro. no/wiki/Performance

However, the configuration recommended by varnish is problematic. The actual operation shows that the configuration of "net.ipv4.tcp_fin_timeout = 3" often causes the page to not open. Moreover, when users use IE6 browser, all the web pages will fail to open after a period of visit to the website. After restarting the browser, it will be normal. Maybe the Internet speed is fast in foreign countries, but we decided to adjust the "net.ipv4.tcp_fin_timeout = 10" in the case of 10s, the 1 cut is normal (actual operation conclusion).

After the modification, execute:

/sbin/sysctl -p /etc/sysctl.conf
/sbin/sysctl -w net.ipv4.route.flush=1

The order came into effect. To be on the safe side, you can also use the reboot system.

Adjust the number of files:

The linux system has to be tuned to increase the number of files allowed to open in order to support large concurrency. The default of 1024 is far from enough.

Execute the command:

Shell code
echo ulimit -HSn 65536 > > /etc/rc.local
echo ulimit -HSn 65536 > > /root/.bash_profile
ulimit -HSn 65536