Example of splitting a very large Redis database

The partner function on the thin load App makes heavy use of the in-memory database Redis. With the rapid growth of the data volume, Redis has expanded rapidly to nearly the size of 12 GB, which is all in a single Redis instance. A single large Redis instance has several disadvantages:

1. First, a large memory machine is needed. Redis is an in-memory database, it needs to put all requirements in memory, need to install 12 GB Redis instances, at least 12 GB memory size of the machine, considering the reserved space for growth, 1 generally requires 12 * 1.5 about 18 GB memory. Another factor to consider is that when Redis is storing hard disk data, fork process needs to consume the same amount of memory, so one instance of redis of 12GB needs around 32 GB memory, which is a very high requirement on the machine and often difficult to meet.

2. Then, Redis can easily become a performance bottleneck. Redis's concurrency model is single-process, single-threaded, which does not take full advantage of multi-core CPU and can be a performance bottleneck when the number of requests is high, or when some requests are processed slowly (such as some large data sort). One way to mitigate even this problem is to create multiple instances of Redis, via multiple Redis connections.

3. In addition, a single large Redis instance will also increase the difficulty of data management, because such a large amount of data, both replication and backup operations are slow, which is easy to impact the online system.

Therefore, it is necessary to divide a single large Redis instance into several small Redis instances.

With Redis's replication mechanism, the Redis instance segmentation can be smooth-handled online with little impact on the system.

The specific operation idea of segmentation is as follows:

1. First, Redis segmentation strategy is planned, usually based on business segmentation. For example, mint partners is based on business segmentation of timeline, user_relationship, other, three instances of Redis. After planning, the Redis program code in the application needs to be modified according to the planning results. Usually, the Redis link with 1 unified is changed to multiple Redis connections, and different businesses use different connections.

2. Then, create multiple copies of Redis through the Redis replication function, let different Redis connections use different copies of Redis, and delete redundant data in Redis copies. To batch delete keys of a mode, you can use the following command shell:

redis-cli KEYS "<pattern>" | xargs redis-cli DEL

Change to the actual mode, e.g

redis-cli KEYS "user:*:followers" | xargs redis-cli DEL

Means to delete user followers data.

Finally, the Redis instance is fully split by switching back and forth and restarting the Redis instance.