Summary of Redis Sentinel implementation of Sentinel mode construction

Redis sentry mode, in popular words, is a "sentry robot". After the corresponding configuration of "sentry robot", the "robot" can work for 7*24 hours. It can automatically help you to do some things, such as monitoring, warning, automatic troubleshooting and so on.

Redis - sentinel profile

Redis-sentinel is the author of Redis, and it took several weeks for antirez to write Redis-ES16en because the Redis cluster was used by various companies, each writing its own cluster management tool.

Redis's Sentinel system is used to manage multiple Redis servers (instance). Redis's Sentinel provides high availability for Redis. Use sentry mode to create an Redis deployment that can cope with various failures without human intervention.

The system performs the following three tasks:

Monitoring (Monitoring) : Sentinel will constantly check to see if your master and slave servers are ok. Alerts (Notification) : When an Redis server being monitored has problems, Sentinel can send notifications to administrators or other applications via API. Automatic failure migration (Automatic failover) : (1) When one master server fails to work properly, Sentinel will start an automatic failure migration operation. It will upgrade one of the slave servers of the failed master servers to the new master server and change the other slave servers of the failed master servers to copy the new master server. (2) When the client tries to connect to the failed master server, the cluster will also return the address of the new master server to the customer service side. Yes, the cluster can use the new master server to replace the failed server.

The distributed nature of sentinel

Redis Sentinel is a distributed system, you can run multiple Sentinel process in an architecture (progress), these processes using gossip protocols (gossip protocols) to receive information on the primary server is offline, and use the voting agreement (agreement protocols) to determine whether to perform automatic fault migration, and choose which as a new primary server from the server.

It is not reliable for a single sentinel process to monitor the redis cluster, and when the sentinel process goes down (sentinel itself has a single point of problem, ES59en-ES60en-ES61en-ES62en) the entire cluster system will not function as expected. So it is necessary to cluster sentinel, which has several benefits:

There are 1 sentinel process down, the redis cluster can still be switched on and off; If there is only one sentinel process, if the process runs incorrectly, or if the network is blocked, then the switch between the redis cluster (single point problem) will not be implemented. If there are multiple sentinel, an redis client can optionally connect to any one of sentinel to get information about the redis cluster

A robust deployment requires at least three sentinel instances.

The three sentinel instances should be placed in the computer or virtual machine on which the client identifies the failure independently. For example, different physical machines or virtual machines with different available areas. This explanation is to build on one machine, and the multi-stage is one reason

background

Recently, I came into contact with the construction of Redis, and simply recorded the pits encountered in the construction process

The overall configuration

192.168.1.100:6379 -> master
192.168.1.101:6379 -> slave
192.168.1.102:6379 -> slave
192.168.1.100:26379 -> sentinel
192.168.1.101:26379 -> sentinel
192.168.1.102:26379 -> sentinel

Set up steps

1. Install redis

#  Unpack the 
tar -xvf /usr/local/redis-3.2.11.tar.gz

mkdir -p /usr/local/redis/bin

cp /usr/local/redis/src/{redis-benchmark,redis-check-aof,redis-check-rdb,redis-cli,redis-sentinel,redis-server,redis-trib.rb} /usr/local/redis/bin

mkdir -p /u01/redis/{6379/{log,data,pid,conf},26379/{log,data,pid,conf}

#  Add environment variables 
echo "export PATH=/usr/local/redis/bin:$PATH" >> /etc/profile
source /etc/profile

2. Configuration redis - 6379

redis basic node configuration as follows, the following configuration cp respectively to three virtual machine/u01 redis / 6379 / conf redis_6379 conf

bind 0.0.0.0
protected-mode no
daemonize yes
pidfile "/u01/redis/6379/pid/redis_6379.pid"
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/u01/redis/6379/log/redis_6379.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/u01/redis/6379/data"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
min-slaves-to-write 1
min-slaves-max-lag 10
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512

Start the service

#  in 3 Each virtual machine executes separately 
redis-server /u01/redis/6379/conf/redis_6379.conf

Establish a master-slave relationship

#  in 192.168.1.101
redis-cli -p 6379 SLAVEOF 192.168.1.100 6379

#  in 192.168.1.102
redis-cli -p 6379 SLAVEOF 192.168.1.100 6379

Check the Replication

192.168.1.101:6379> info replication
# Replication
role:master
connected_slaves:2
min_slaves_good_slaves:2
slave0:ip=192.168.1.102,port=6379,state=online,offset=9577826,lag=1
slave1:ip=192.168.1.103,port=6379,state=online,offset=9577965,lag=0
master_repl_offset:9577965
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:8529390
repl_backlog_histlen:1048576

192.168.1.102:6379> info replication
# Replication
role:slave
master_host:192.168.1.101
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:9600220
slave_priority:100
slave_read_only:1
connected_slaves:0
min_slaves_good_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

192.168.1.103:6379> info replication
# Replication
role:slave
master_host:192.168.1.101
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:9612675
slave_priority:100
slave_read_only:1
connected_slaves:0
min_slaves_good_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

3. The configuration sentinel - 6379

sentinel basic node configuration as follows, the following configuration cp respectively to three virtual machine/u01 redis / 26379 / conf sentinel_26379 conf

sentinel monitor mymaster is monitored after the master node in redis, which is 192.168.1.100, so this file is the same on all three machines

port 26379
bind 0.0.0.0
daemonize yes
protected-mode no
dir "/u01/redis/26379/tmp"
logfile "/u01/redis/26379/log/sentinel_26379.log"
sentinel monitor mymaster 192.168.1.100 6379 1

Waiting to start after the observation/u01 redis / 26379 / conf/sentinel_26379 conf file changes

View sentinel status using info sentinel

redis-cli -h 192.168.1.100 -p 26379 info sentinel

# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=zhuanche01,status=ok,address=192.168.1.100:6379,slaves=2,sentinels=3

conclusion

When I built it, I encountered the problem that sentinel on 192.168.1.101 and 192.168.1.102 made an error during the first period after startup, and later I found that master was not monitored
Then there is the problem of log

Write more notes next year. Your memory will get worse as you get older.