Redis Cluster complete steps for adding and removing
- 2020-06-01 11:15:33
- OfStack
preface
Recently, I learned Redis, and found that Redis is quite interesting. Today, I tested adding, deleting nodes, and redistributing slot in clusters. A deeper understanding of redis's game rules. There are many steps, but more details. Let's take a look at the details.
Environmental explanation:
I tested it on one Centos 6.9, with each redis node distinguished by port number. For each redis in this article, I only represent it with a port number.
~~~~Master Node~~~~~
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
~~~~Slave Node~~~~~
172.16.32.116:8000
172.16.32.116:8001
172.16.32.116:8002
~~~~ Toss about Node~~~~~
172.16.32.116:9000
172.16.32.116:9001
1. Create an redis cluster
Note: for more redis cluster creation, see
Redis Cluster cluster deployment setup
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.2. Check the status of the cluster
# ./redis-trib.rb check 172.16.32.116:7000
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) slave
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) slave
replicates 273107e5ac994d675749be0979556e761274bb93
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) slave
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ add node ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
3. Add new node redis-trib.rb add-node add new node name and original cluster node name
# ./redis-trib.rb add-node 172.16.32.116:9000 172.16.32.116:7000
>>> Adding node 172.16.32.116:9000 to cluster 172.16.32.116:7000
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) slave
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) slave
replicates 273107e5ac994d675749be0979556e761274bb93
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) slave
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...4. Check the current cluster status, 9000 is an empty Master
# ./redis-cli -p 9000 cluster nodes
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505321254767 1 connected 0-5460
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505321250759 2 connected 5461-10922
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505321251761 3 connected 10923-16383
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505321255769 3 connected
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505321253765 1 connected
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505321256771 2 connected5. Allocate slot for 9000, solt of redis is fixed, there are only 16384 of slot, which can only be obtained from other nodes and then allocated to 9000
# ./redis-trib.rb reshard 172.16.32.116:9000
>>> Performing Cluster Check (using node 172.16.32.116:9000)
M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000
slots: (0 slots) master
0 additional replica(s)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) slave
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) slave
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) slave
replicates 273107e5ac994d675749be0979556e761274bb93
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?
What is the receiving node ID? 364ae8322ab2627e25b05d45b702448c74afad10 Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
Ready to move 300 slots.
Source nodes:
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
Destination node:
M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000
slots: (0 slots) master
0 additional replica(s)
Resharding plan:
Moving slot 5461 from 273107e5ac994d675749be0979556e761274bb93
Moving slot 5469 from 273107e5ac994d675749be0979556e761274bb93
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 5461 from 172.16.32.116:7001 to 172.16.32.116:9000:6. It can be seen that the slot assigned by 9000 is 0-98 5461-5561 10923-11021
# ./redis-cli -p 9000 cluster nodes
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505324905062 1 connected 99-5460
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505324910075 2 connected 5562-10922
364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 myself,master - 0 0 7 connected 0-98 5461-5561 10923-11021
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505324908070 3 connected
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505324911077 1 connected
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505324902057 3 connected 11022-16383
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505324909073 2 connected
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ 9000 into slave ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
7. We hope to change 9000 into 7000 slave, but due to the failure of slot, we need to transfer slot first
# redis-cli -c -p 9000 cluster replicate a0b91f48e933c1f1d427c54917ce970bd25d29f8
(error) ERR To set a master the node must be empty and without assigned slots.8. Deleting nodes is also not allowed, in short, as long as slot is on it. redis is not going to let you delete it, and it's going to require human intervention, rebalance, slot
# ./redis-trib.rb del-node 172.16.32.116:9000 364ae8322ab2627e25b05d45b702448c74afad10
>>> Removing node 364ae8322ab2627e25b05d45b702448c74afad10 from cluster 172.16.32.116:9000
[ERR] Node 172.16.32.116:9000 is not empty! Reshard data away and try again.9. Redistribute 9000 slot to 7000
# ./redis-trib.rb reshard 172.16.32.116:9000 <<<<<< redistribution slot
>>> Performing Cluster Check (using node 172.16.32.116:9000)
M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000
slots:0-98,5461-5561,10923-11021 (299 slots) master
0 additional replica(s)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:99-5460 (5362 slots) master
1 additional replica(s)
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5562-10922 (5361 slots) master
1 additional replica(s)
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) slave
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) slave
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:11022-16383 (5362 slots) master
1 additional replica(s)
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) slave
replicates 273107e5ac994d675749be0979556e761274bb93
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 300 <<<9000 All nodes that need to be migrated
What is the receiving node ID? a0b91f48e933c1f1d427c54917ce970bd25d29f8 <<<<<<7000 the ID
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:364ae8322ab2627e25b05d45b702448c74afad10 <<<<<<9000 the ID
Source node #2:done
Ready to move 300 slots.
Source nodes:
M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 <<<<<<Source nodes
slots:0-98,5461-5561,10923-11021 (299 slots) master
0 additional replica(s)
Destination node:
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 <<<<<<<Destination node
slots:99-5460 (5362 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 0 from 364ae8322ab2627e25b05d45b702448c74afad10
.........
Moving slot 11021 from 364ae8322ab2627e25b05d45b702448c74afad10
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 0 from 172.16.32.116:9000 to 172.16.32.116:7000:
Moving slot 1 from 172.16.32.116:9000 to 172.16.32.116:7000:
........10. On inquiry, you can see that 9000 has no slot
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.11. Execute the command again and turn 9000 into 7000 slave, successful
# redis-cli -c -p 9000 cluster replicate a0b91f48e933c1f1d427c54917ce970bd25d29f8
OK12. Check the status,9000 has become 7000 slave
# ./redis-cli -p 9000 cluster nodes
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505329564286 8 connected 0-5561 10923-11021
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505329561281 2 connected 5562-10922
364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 myself,slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 0 7 connected <<<<<<<<<<<<<<
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505329558274 3 connected
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505329554266 8 connected
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505329563285 3 connected 11022-16383
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505329562283 2 connected13. Node 9000 was deleted, which was successfully deleted
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.14. Connecting 9000, I found that shutdown has been found and cannot be connected
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.15. Check the cluster status, 9000 is gone
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ again started 9000, found that different ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
16. Start 9000 again and find one interesting thing. Check 7000 above, the cluster is no longer 9000
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.17. However, if you look at 9000, you can still see the information for the entire cluster.
Note that in the process of deleting nodes, only 9000 information is deleted in the original cluster. However, the information of 9000 itself has not been deleted, and all information is still retained, only 9000 instances are closed.
# ./redis-cli -p 9000 cluster nodes
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505329902003 8 connected 0-5561 10923-11021
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505329903006 2 connected 5562-10922
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505329906013 3 connected
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505329908019 2 connected
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505329904008 8 connected
364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 myself,slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 0 7 connected
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505329907016 3 connected 11022-1638318. All information of 9000 is recorded in nodes.conf in its own directory
# more nodes.conf
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505329544244 8 connected 0-5561 10923-11021
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505329542241 2 connected 5562-10922
364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 myself,slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 0 7 connected
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505329541239 3 connected
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505329545246 8 connected
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505329543242 3 connected 11022-16383
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505329546248 2 connected
vars currentEpoch 8 lastVoteEpoch 019. Go to other nodes such as 7001 and check that nodes.conf no longer has the information of 9000
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.32.116:7000
172.16.32.116:7001
172.16.32.116:7002
Adding replica 172.16.32.116:8000 to 172.16.32.116:7000
Adding replica 172.16.32.116:8001 to 172.16.32.116:7001
Adding replica 172.16.32.116:8002 to 172.16.32.116:7002
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
replicates 273107e5ac994d675749be0979556e761274bb93
S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 172.16.32.116:7000)
M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000
slots:0-5460 (5461 slots) master
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5461-10922 (5462 slots) master
M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002
slots:10923-16383 (5461 slots) master
M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000
slots: (0 slots) master
replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8
M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001
slots: (0 slots) master
replicates 273107e5ac994d675749be0979556e761274bb93
M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002
slots: (0 slots) master
replicates 88fe075375295b59eabe69fa1438ed7c7c314f43
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ again to add 9000, and 9001 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
20. Rejoin 9000 and 9001 into the cluster and start the next twist.
Note: files outside redis.conf under 9000 and 9001 need to be cleared and restarted. Then you can join the cluster again. Otherwise you will encounter errors:
[ERR] Node 172.16.32.116:9001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.21. Join a cluster
./redis-trib.rb add-node 172.16.32.116:9000 172.16.32.116:7002
./redis-trib.rb add-node 172.16.32.116:9001 172.16.32.116:700222. Now there are two empty Master nodes
# ./redis-trib.rb check 172.16.32.116:9001
>>> Performing Cluster Check (using node 172.16.32.116:9001)
M: c4ba7a1f537ac66076791461d6af9012741fee74 172.16.32.116:9001
slots: (0 slots) master
0 additional replica(s)
M: dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 172.16.32.116:9000
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.23. Redistribute 100 slot to 9000, actually redis is very smart, I'm connecting 9001, but when you allocate it, it will ask you who receiving node is and Source node is.
# ./redis-trib.rb reshard 172.16.32.116:9001
>>> Performing Cluster Check (using node 172.16.32.116:9001)
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 100
What is the receiving node ID? dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 <<<<<<<<<receiving node ID
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:aeb684429d220c0fd1392574d193cc1ae7577782 <<<<<<<Source node I choose 8000
*** The specified node is not known or is not a master, please retry. <<<<<<< However, there was no cheating redis It finds that this is slave , there is no slot It can be provided.
Source node #1:273107e5ac994d675749be0979556e761274bb93 <<<<<<<<Source node Specify again as 7001 Let's start distributing
Source node #2:done
Ready to move 100 slots.
Source nodes:
M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001
slots:5562-10922 (5361 slots) master
1 additional replica(s)
Destination node:
M: dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 172.16.32.116:9000
slots: (0 slots) master
0 additional replica(s)
Resharding plan:
Moving slot 5562 from 273107e5ac994d675749be0979556e761274bb93
Moving slot 5563 from 273107e5ac994d675749be0979556e761274bb9324. Check the allocation. redis is still smart and flexible.
# redis-cli -p 7001 cluster nodes
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 myself,master - 0 0 2 connected 5662-10922
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505330856605 8 connected 0-5561 10923-11021
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505330853598 8 connected
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505330860611 6 connected
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505330859608 5 connected
c4ba7a1f537ac66076791461d6af9012741fee74 172.16.32.116:9001 master - 0 1505330862615 9 connected <<<<<<<<<<<<<<<<<<<<< It was still empty
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505330861612 3 connected 11022-16383
dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 172.16.32.116:9000 master - 0 1505330858607 10 connected 5562-5661 <<<<<<<<<<<<<<<<<<<<< from 7001 Want to come to 100 a slot25. Add 9001 to the slave node redis-cli-p as 9000 < slave IP:port > cluster nodes < Master ID no. >
# redis-cli -p 9001 cluster nodes dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516
273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 myself,master - 0 0 2 connected 5662-10922
a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505331457798 8 connected 0-5561 10923-11021
aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505331454791 8 connected
3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505331456795 6 connected
a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505331458799 10 connected
c4ba7a1f537ac66076791461d6af9012741fee74 172.16.32.116:9001 slave dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 0 1505331459801 10 connected
88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505331455793 3 connected 11022-16383
dbf78b73f2ab9e37cbf31abbc2beb3d5413d5516 172.16.32.116:9000 master - 0 1505331453788 10 connected 5562-5661
After a lot of fiddling, redis added, deleted, redistributed slot and so on, it's all been tested.
There are a lot of command output departments in the middle, which seems a bit repetitive. But for better reading comprehension, observe the changes at each step. It's a little bit easier to look up later.
After all, old age has a bad memory. A lot of blog that I have written, when I look back, I find some steps that are not easy to understand. Even though my blog is based on the results of my own tests, I still forget.
conclusion