Use the pipeline batch insert method in the Redis cluster

Due to the need to use batch insert function in the project, Redis batch insert can be found on the Internet to use pipeline to insert efficiently. The sample code is as follows:

String key = "key";
Jedis jedis = new Jedis("xx.xx.xx.xx");
Pipeline p = jedis.pipelined();
List<String> myData = .... // The list of data to insert 
for(String data: myData){
  p.hset(key, data);
}
p.sync();
jedis.close();

The actual problem is that the Redis used on the project is clustered and the class is JedisCluster instead of Jedis. A search of JedisCluster's documentation shows that there is no pipelined() method for retrieving Pipeline objects like Jedis1.

Google hit 1 and found a solution.

The Redis cluster specification states that the Redis cluster is divided into 16,384 slots (slot) and the maximum number of nodes in the cluster is 16,384. Each master node is responsible for processing one of the 16,384 hash slots. When we say that a cluster is in a "stable" (stable) state, we mean that the cluster is not performing a reconfiguration (reconfiguration) operation and that each hash slot is handled by only one node.

Therefore, we can know the slot number corresponding to this key according to the key to be inserted, and then find the slot number corresponding to Jedis from the cluster. The specific implementation is as follows

// So I'm going to initialize it jedis cluster,  How to get HostAndPort The collection code is not written 

Set<HostAndPort> nodes = .....

JedisCluster jedisCluster = new JedisCluster(nodes);



Map<String, JedisPool> nodeMap = jedisCluster.getClusterNodes();

String anyHost = nodeMap.keySet().iterator().next();

//getSlotHostMap The method is shown below 

TreeMap<Long, String> slotHostMap = getSlotHostMap(anyHost); 

  private static TreeMap<Long, String> getSlotHostMap(String anyHostAndPortStr) {
    TreeMap<Long, String> tree = new TreeMap<Long, String>();
    String parts[] = anyHostAndPortStr.split(":");
    HostAndPort anyHostAndPort = new HostAndPort(parts[0], Integer.parseInt(parts[1]));
    try{
      Jedis jedis = new Jedis(anyHostAndPort.getHost(), anyHostAndPort.getPort());
      List<Object> list = jedis.clusterSlots();
      for (Object object : list) {
        List<Object> list1 = (List<Object>) object;
        List<Object> master = (List<Object>) list1.get(2);
        String hostAndPort = new String((byte[]) master.get(0)) + ":" + master.get(1);
        tree.put((Long) list1.get(0), hostAndPort);
        tree.put((Long) list1.get(1), hostAndPort);
      }
      jedis.close();
    }catch(Exception e){
      
    }
    return tree;
  }

The above steps can be done at initialization time. Instead of calling each time, define nodeMap and slotHostMap as static variables.

// Get slot Numbers 

int slot = JedisClusterCRC16.getSlot(key); 

// Gets the corresponding Jedis object 

Map.Entry<Long, String> entry = slotHostMap.lowerEntry(Long.valueOf(slot));

Jedis jedis = nodeMap.get(entry.getValue()).getResource();

It is recommended that this step be encapsulated as a static method, such as public static Jedis getJedisByKey(String key), which means that the Jedis object corresponding to key is obtained from key in the cluster.

This is done through jedis.pipelined () above; You can do batch inserts.

Note: This method is from Google and I haven't found anything wrong with it so far. If any of you find anything wrong, please feel free to ask.