springboot integrates the operation flow of redis modifying partition

springboot Integrate redis Modify Partition

Origin of the problem

Recently, springboot is used to integrate redis, a system dynamic data source is connected to different databases, and redis is used for caching, so it is necessary to cache the data of different databases to different partitions of redis, that is, different libraries.

Old version solution

The old version here is before 2.0, and the 1.5. 9 I used is ok.

The configuration class of redis is not posted here, and there are many on the Internet.

1. Modify with JedisConnectionFactory

@Autowired
JedisConnectionFactory jedisConnectionFactory;
jedisConnectionFactory.setDatabase(database);

2. Modify with redisTemplate

redisTemplate.getConnectionFactory().getConnection().select(database);

The above two methods do not need to add bean to the redis configuration class

New version solution

The new version here refers to after 2.0, and I use 2.0. 3

The following bean needs to be added to the redis configuration class

    @Bean
    RedisStandaloneConfiguration redisStandaloneConfiguration() {
        RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration();
        redisStandaloneConfiguration.setHostName("localhost");
        redisStandaloneConfiguration.setPort(6379);
        redisStandaloneConfiguration.setDatabase(0);
        return redisStandaloneConfiguration;
    }
    @Bean
    JedisConnectionFactory jedisConnectionFactory(RedisStandaloneConfiguration redisStandaloneConfiguration) {
        //redisStandaloneConfiguration.setPassword(RedisPassword.of(password));
        JedisClientConfiguration.JedisClientConfigurationBuilder jedisClientConfiguration = JedisClientConfiguration.builder();
        jedisClientConfiguration.connectTimeout(Duration.ofMillis(0));//  connection timeout
        JedisConnectionFactory factory = new JedisConnectionFactory(redisStandaloneConfiguration,
                jedisClientConfiguration.build());
        return factory;
    }

Modify with RedisStandaloneConfiguration

@Autowired
RedisStandaloneConfiguration redisStandaloneConfiguration;
redisStandaloneConfiguration.setDatabase(database);

redis partition

How data is distributed across multiple Redis instances

Partitioning is to distribute your data over multiple Redis instances so that each instance contains only one part of the data.

Why is partitioning useful

The Redis partition has two main goals:

It allows for larger databases, using the sum of memory of many computers. If you don't partition, you will be limited to the memory of a single computer. It allows computing power to be extended to multiple cores and computers, and network bandwidth to multiple computers and network adapters.

Suppose we have four Redis instances (R0, R1, R2, R3) with many key representing users, such as user: 1, user: 2,..., etc., then we have many ways to store one key.

The simplest way is to partition by scope, that is, to allocate data to a specified Redis instance according to the mapping scope of the object. For example, if we specify that ID is 0 ~ 10000, it will be divided into R0, 10001 ~ 20000 to R1, and so on. This is possible, but one drawback is that one table is required to maintain this mapping. This table needs to be managed, and one for every key, so range partitioning in Redis is generally undesirable because it is much less efficient than other partitioning methods.

In addition to range partitioning, another method is hash partitioning (hash partitioning):

Step 1, take key and apply hash function to convert it to 1 number. For example, if key is foobar and the hash function crc32, crc32 (foobar) will output 93024922.

Step 2, using modulo operation (modulo) on this number to convert it to a direct number from 0 to 3, so that this number can be mapped to one of my four Redis instances. For example, 93024922% 4 = 2, so foobar should be stored on the R2 instance.

(PS: First, hash key to get a number, and then take the modulo on this number to determine which instance the final data should be stored on.)

There are many ways to partition. From the above two examples, you should understand. A high-level form of hash partitioning, called a 1-hash, is implemented by several Redis clients and proxies.

Different partition implementations

Client partition: For a given key, the client directly selects the correct node for reading and writing. Many Redis clients implement client partitioning.

Proxy Partition: The client sends a request to a proxy, which communicates with Redis, and the proxy will select the correct Redis instance according to our configuration.

Query routing: You can send your query to any one Redis instance, which will redirect your query to the correct server.

(PS: For a given key, the job of the partition is to select the correct Redis instance, which can be done by the client, agent, or Redis instance.)

Shortcomings of zoning

1. Operations involving multiple key are generally not supported. For key mapped to two different instances of Redis, you cannot insert on these two instances.

2. Operations involving multiple key cannot use Redis transactions

3. Partition granularity is key, so it is impossible to segment data from a single very large key (for example, a very large sorted set)

4. When using partitions, data processing will be more complicated. For instances, you must process multiple RDB/AOF files. In order to back up data, you need to aggregate persistent files from multiple instances and hosts.

5. Adding and deleting capacity (space) becomes more complicated. For example, an Redis cluster supports transparent data rebalancing for adding and removing nodes at run time, but other systems such as client partitions and agents do not. However, a technique called pre-slicing is helpful in this respect.

Data storage or cache?

When an Redis is used as a data store, a given key must always map to the same Redis instance. When used as a cache, it is not a big problem if a given node is unavailable.

If the preferred node for a given key is not available, the 1-to-hash implementation is usually able to switch to another node. Similarly, if a new node is added, some of the new keys will begin to be stored on the new node.

If Redis is used as cache, it is easy to scale with 1 hash. If Redis is used as storage, a fixed keys-to-nodes mapping is used, so the number of nodes must be fixed and cannot be changed. Otherwise, you need a system that can rebalance key between nodes, and the current Redis cluster can do this.