Of Stack

Problems and Solutions of Vmware Deploying Nginx+KeepAlived Cluster Dual Main Architecture

  • 2021-11-02 03:41:01
  • OfStack

Preface

Using nginx as load balancing, as the front end or middle layer of the architecture, with the increasing number of visits, it is necessary to make a high-availability architecture for load balancing, use keepalived to solve single-point risk, and quickly switch to backup server when nginx goes down.

Solutions to Possible Problems in Vmware Network Configuration

Start VMware DHCP Service And VMware NAT Service Two services Turn on network sharing in the network adapter, allow other networks to check Save, and restart the virtual machine

Installation

Node deployment

节点 地址 服务
centos7_1 192.168.211.130 Keepalived+Nginx
centos7_2 192.168.211.131 Keepalived+Nginx
centos7_3 192.168.211.132 Redis服务器
web1（物理机） 192.168.211.128 FastApi+Celery
web2（物理机） 192.168.211.129 FastApi+Celery

Configuration of web

web1 starts python http server 


vim index.html

<html>
<body>
<h1>Web Svr 1</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &

web2 Start python http Server 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &

Close the firewall 


firewall-cmd --state
systemctl stop firewalld.service
systemctl disable firewalld.service

Browser access is now normal and the page displays Web Svr 1 and 2

centos1 and 2 Install Nginx

First configure Alibaba Cloud's source 


mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

Install dependency packages 


yum -y install gcc
yum install -y pcre pcre-devel
yum install -y zlib zlib-devel
yum install -y openssl openssl-devel

Download nginx and unzip it 


wget http://nginx.org/download/nginx-1.8.0.tar.gz
tar -zxvf nginx-1.8.0.tar.gz

Installing nginx 


cd nginx-1.8.0
./configure --user=nobody --group=nobody --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_gzip_static_module --with-http_realip_module --with-http_sub_module --with-http_ssl_module
make
make install
cd /usr/local/nginx/sbin/
#  Check the configuration file 
./nginx -t
#  Start nginx
./nginx

Open nginx access 


firewall-cmd --zone=public --add-port=80/tcp --permanent
systemctl restart firewalld.service

Visit both 130 and 131 to see the home page of nginx.

Create an nginx startup file

You need to create the nginx startup file in the init. d folder. This automatically starts Nginx each time the server restarts the init process. 


cd /etc/init.d/
vim nginx

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemin
#
# chkconfig:   - 85 15
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# pidfile:     /var/run/nginx.pid
# user:        nginx

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"

lockfile=/var/run/nginx.lock

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

To verify the configuration file, enter the following commands in turn 


chkconfig --add nginx
chkconfig --level 345 nginx on

Add execute permission to this file 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
0

Start the Nginx service 


service nginx start
service nginx status
service nginx reload

Nginx Reverse Proxy, Load Balancing (centos_1)

Modify the nginx. conf configuration file to remove the annotated code 


cd /usr/local/nginx/conf/
mv nginx.conf nginx.conf.bak
egrep -v '^#' nginx.conf.bak
egrep -v '^#|^[ ]*#' nginx.conf.bak
egrep -v '^#|^[ ]*#|^$' nginx.conf.bak 
egrep -v '^#|^[ ]*#|^$' nginx.conf.bak >> nginx.conf
cat nginx.conf

The output is as follows 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
3

Reload the nginx configuration 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
4

Configure nginx reverse proxy, load balancing 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
5

Now restart nginx 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
6

The websvr name can be customized to indicate the meaning of these servers. That is, you only need to add upstream websvr And proxy_pass Load balancing can be achieved.

When you visit 130 now, Web Svr 1 and Web Svr 2 will be switched on the page, and the server will be selected according to the weight. The higher the weight value, the higher the weight, that is, the page will be refreshed repeatedly, with an average of Web Svr 2 appearing twice and Web Svr 1 appearing once.

Up to now, it is still impossible to achieve high availability. Although web service can do this and single point of failure can be handled in this way, if nginx service fails, the whole system is basically inaccessible, so multiple Nginx are needed to guarantee it.

Multiple Nginx work together, and Nginx is highly available "dual-machine master-slave mode"

In 131 A new nginx service is added to the server (centos2), which is the same as the previous configuration, and only needs to modify nginx. conf 


worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

        upstream websvr {
        server 192.168.211.128:8001  weight=1;
        server 192.168.211.129:8001  weight=2;
    }

    server {
        listen       80;
        server_name  192.168.211.131;
        location / {
            proxy_pass http://websvr;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

#  Reload nginx
sbin/nginx -s reload

Similar results can be obtained by visiting http://192.168. 211.130/now, which is similar to http://192.168. 211.131/.

The IP of these two Nginx servers are different, so what can I do to make these two nginx servers work together? This requires keepalived.

Install the software, and install both centos at the same time 


yum install keepalived pcre-devel  -y

Configuring keepalived

Both are backed up 


vim index.html

<html>
<body>
<h1>Web Svr 2</h1>
</body>
</html>

nohup python -m SimpleHTTPServer 8080 > running.log 2>&1 &
9

centos_1 Configure Keepalived-MASTER 


[root@localhost keepalived]# cat keepalived.conf
! Configuration File for keepalived

global_defs {
    script_user root
	enable_script_security
}

vrrp_script chk_nginx {
    #  Specify monitoring scripts, detect nginx Is the service running normally 
    script "/etc/keepalived/chk_nginx.sh"
    #  Specify the monitoring time, and every 10s Execute 1 Times 
    interval 10
    #  Priority change caused by script result, detection failed (script returned non- 0 ), priority  -5
    # weight -5
    # #  Continuous detection 2 A failure is a true failure. Can use weight Reduce priority ( 1-255 Between) 
    # fall 2
    #  Detection 1 A second success is a success. But the priority is not modified 
    # rise 1
}

vrrp_instance VI_1 {
	#  Specify keepalived The host is set to the role of MASTER The standby machine is set to BACKUP
    state BACKUP
	#  Specify HA Monitor the interface of the network. centos7 Use  ip addr  Get 
    interface ens33
	#  Primary and standby virtual_router_id Must 1 Sample, which can be set to IP Posterior 1 Group: must be between 1 & 255
    virtual_router_id 51
	#  Priority value, in the same 1 A vrrp_instance Under,  MASTRE 1 Must be higher than  BAUCKUP , MASTER After recovery, BACKUP Automatic handover 
    priority 90
	# VRRP  The broadcast cycle is seconds. If the broadcast is not detected, it is considered that the service is dead and the main and standby will be switched 
    advert_int 1
	#  Set the authentication type and password. Master and slave must 1 Sample 
    authentication {
		#  Settings vrrp Validation types, mainly including PASS And AH Two kinds 
        auth_type PASS
		#  Encrypted password, two servers 1 Be sure 1 Sample, in order to communicate normally 
        auth_pass 1111
    }
	track_script {
        #  Services that perform monitoring, reference VRRP Script, that is, in the  vrrp_script  Partially specified name. Run them periodically to change priority 
        chk_nginx
    }
    virtual_ipaddress {
		# VRRP HA  Virtual address   If there are multiple VIP Continue to fill in the new line 
        192.168.211.140
    }
}

Send the configuration file to 131 Node 


scp /etc/keepalived/keppalived.conf 192.168.211.131:/etc/keepalived/keepalived.conf

For 131 Nodes only need to be modified 


state BACKUP
priority 90

Main keepalived configuration monitoring script chk_nginx. sh

Create 1 script for execution in keepalived 


vi /etc/keepalived/chk_nginx.sh

#!/bin/bash
#  Check to see if there are  nginx Process   Assign a value to a variable counter
counter=`ps -C nginx --no-header |wc -l`
#  If no process is worth  0
if [ $counter -eq 0 ];then
    #  Try to start nginx
    echo "Keepalived Info: Try to start nginx" >> /var/log/messages
    /etc/nginx/sbin/nginx
    sleep 3
    if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
        #  Output day-to-day system message 
        echo "Keepalived Info: Unable to start nginx" >> /var/log/messages
        #  If it hasn't started yet, it ends  keepalived  Process 
        # killall keepalived
        #  Or stop 
        /etc/init.d/keepalived stop
        exit 1
    else
        echo "Keepalived Info: Nginx service has been restored" >> /var/log/messages
        exit 0
    fi
else
    #  The state is normal 
    echo "Keepalived Info: Nginx detection is normal" >> /var/log/messages;
    exit 0
fi

Next, grant execute permission and test 


chmod +x chk_nginx.sh
./chk_nginx.sh

Restart keepalived on both sides 


systemctl restart keepalived
systemctl status keepalived

Access at this time .140 It can also be displayed normally, that is, the bound IP is successful. Before execution, you can view the output log in messages in real time with the following command 


tail -f /var/log/messages 

#  If nginx Shut down 
Keepalived Info: Try to start nginx
Keepalived Info: Nginx service has been restored
# nginx Open normally 
Keepalived Info: Nginx detection is normal

When nginx detection is normal, it will return 0; The detection is gone and returns 1, but keepalived does not seem to detect this return value to realize the transfer, but detects whether the keepalived service exists to release the local VIP and finally transfer the virtual IP to another server.

Reference article

https://www.jianshu.com/p/7e8e61d34960
https://www.cnblogs.com/zhangxingeng/p/10721083.html

Related articles: