Process resolution for Nginx high availability solution in production environment

Preparation:

192.168.16.128

192.168.16.129

Two virtual machines. Installed Nginx

Install Nginx

Update yum source file:

rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

Install Nginx:

yum -y install nginx

Operation command:

systemctl start nginx; # Start the Nginx
systemctl stop nginx; # stop Nginx

What is high availability?

High availability HA (High Availability) is one of the factors that must be considered in designing a distributed system architecture. If a system can provide services directly, the availability is 100 percent, but things can happen. So we can only reduce the failure of the service as much as possible.

The problem solved?

In the production environment, Nginx is often used as a reverse agent to provide external services. However, Nginx will inevitably encounter failures in one day, such as server outage. When Nginx goes down, all externally provided interfaces become inaccessible.

Although we can't guarantee 100% availability of our servers, we have to find a way to avoid this tragedy. Today we use keepalived to implement Nginx

High availability.

Dual machine hot standby plan

This scheme is the most common high availability scheme in domestic enterprises. In fact, double machine hot standby means that one server is providing service and the other one is the standby state of a certain service. When one server is unavailable, another server will take its place.

What is keepalived?

Keepalived software was originally designed for LVS load balancing software to manage and monitor the status of various service nodes in the LVS cluster system. Later, it added the function of VRRP (Virtual Router Redundancy Protocol, virtual router redundancy protocol), which can realize high availability. Therefore, in addition to being able to manage the LVS software, Keepalived can also be used as a high-availability solution software for other services such as Nginx, Haproxy, MySQL, and so on

Failover mechanism

Failover over between the highly available Keepalived services is achieved through VRRP.

When Keepalived service work, the main Master node will continue to send for node (multicast) heartbeat message, to tell themselves alive for Backup node, node failure occurs when the main Master, cannot send a heartbeat message, for node is unable to continue to test coming autonomous heartbeat Master node, then calls itself to take over the program, to take over the main Master node IP resources and services. When the primary Master node is restored, the standby Backup node will release the IP resources and services it took over when the primary node failed, and revert to its original standby role.

The implementation process

Install keepalived

You can install it directly in yum mode, which will install the dependencies automatically:

yum -y install keepalived

Modify the host (192.168.16.128) keepalived configuration file

The production profile installed in yum mode is under /etc/keepalived:

vi keepalived.conf

keepalived.conf:

# Test script 
vrrp_script chk_http_port {
 script "/usr/local/src/check_nginx_pid.sh" # The heartbeat executes the script to detect nginx Whether to start the 
 interval 2       # (detect the interval of script execution in seconds) 
 weight 2       # The weight 
}
#vrrp  Instance definition section 
vrrp_instance VI_1 {
 state MASTER   #  The specified keepalived The role of, MASTER Give priority to, BACKUP For the case 
 interface ens33   #  The current in vrrp Communication network interface card ( The current centos The network card )  with ifconfig Check your specific network card 
 virtual_router_id 66 #  Virtual route number, master and slave 1 straight 
 priority 100   #  Priority, the higher the value, the higher the priority of getting the request processed 
 advert_int 1   #  Check the interval, by default 1s(vrrp The number of seconds of multicast cycles )
 # Authorized to access 
 authentication {
  auth_type PASS # Set the authentication type and password, MASTER and BACKUP You must use the same password to communicate normally 
  auth_pass 1111
 }
 track_script {
  chk_http_port   # (call detection script) 
 }
 virtual_ipaddress {
  192.168.16.130   #  Define the virtual ip(VIP) , can be set more, each line 1 a 
 }
}

virtual_ipaddress allows you to configure vip and access services online via vip.

interface needs to be set according to the server network card

The same configuration is required for the authentication configuration to grant access to the backup

Modify the backup (192.168.16.129) keepalived configuration file

keepalived.conf:

# Test script 
vrrp_script chk_http_port {
 script "/usr/local/src/check_nginx_pid.sh" # The heartbeat executes the script to detect nginx Whether to start the 
 interval 2       # (detect the interval between script execution) 
 weight 2       # The weight 
}
#vrrp  Instance definition section 
vrrp_instance VI_1 {
 state BACKUP      #  The specified keepalived The role of, MASTER Give priority to, BACKUP For the case 
 interface ens33      #  The current in vrrp Communication network interface card ( The current centos The network card )  with ifconfig Check your specific network card 
 virtual_router_id 66    #  Virtual route number, master and slave 1 straight 
 priority 99       #  Priority, the higher the value, the higher the priority of getting the request processed 
 advert_int 1      #  Check the interval, by default 1s(vrrp The number of seconds of multicast cycles )
 # Authorized to access 
 authentication {
  auth_type PASS # Set the authentication type and password, MASTER and BACKUP You must use the same password to communicate normally 
  auth_pass 1111
 }
 track_script {
  chk_http_port     # (call detection script) 
 }
 virtual_ipaddress {
  192.168.16.130     #  Define the virtual ip(VIP) , can be set more, each line 1 a 
 }
}

Detection script:

#!/bin/bash
# detection nginx Is it activated or not 
A=`ps -C nginx --no-header |wc -l`  
if [ $A -eq 0 ];then # if nginx Start without starting nginx      
  systemctl start nginx    # restart nginx
  if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then #nginx If the restart fails, stop keepalived Service, conduct VIP transfer 
    killall keepalived     
  fi
fi

Script authorization :chmod 775 check_nginx_pid.sh

Note: the script must be authorized, otherwise there is no access, here we two servers to execute, VIP(virtual_ipaddress:192.168.16.130), We access the service directly through vip in the production environment.

Simulated nginx failure:

Modify the html page of Nginx accessed by both servers by default as a distinction.

The first visit 192.168.16.130 , through vip, the page is displayed 192.168.16.128 ; Indicates that the service is currently provided by the primary server.

At this time 192.168.16.128 Master server execute command:

systemctl stop nginx; # stop nginx

Visit vip( 192.168.16.130 ) at this point, the page still displays: 192.168.16.128 , this is the script inside the automatic restart.

Now I'm gonna go ahead and 192.168.16.128 Server down, access vip( 192.168.16.130 ) the page is now displayed 192.168.16.129 At this time keepalived With automatic failover, a set of high availability solutions for an enterprise production environment is set up.

keepalived In still have a lot of function for instance: mailbox reminds to wait a moment, do not operate, can go to official website to see a document.