Java batch read write to Redis using Pipeline (hmsethgetall)

Generally, after the Redis Client terminal sends a request, it will block and wait for the Redis server to process. After the Redis server processes the request, it will return the result to Client via the response message.

This feels a bit similar to Scan for HBase, which is usually the Client side that gets every record one time and makes every RPC call to the server.

In Redis, is there something like HBase Scanner Caching, one request, multiple records?

Yes, this is Pipline. Official introduction http: / / redis io/topics/pipelining

With pipeline, we can save a lot of time wasted in network latency when there are a large number of operations. It is important to note that the command is packaged with pipeline, and redis must cache the processing results of all commands before processing them. The more commands you pack, the more memory your cache consumes. So it's not as if the more commands you have, the better.

When using Pipeline for bulk reads and writes to Redis, there is a significant performance improvement.

Java tested 1:

package com.lxw1234.redis;

import java.util.HashMap;
import java.util.Map;
import java.util.Set;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import redis.clients.jedis.Response;


public class Test {
 
 public static void main(String[] args) throws Exception {
 Jedis redis = new Jedis("127.0.0.1", 6379, 400000);
 Map<String,String> data = new HashMap<String,String>();
 redis.select(8);
 redis.flushDB();
 //hmset 
 long start = System.currentTimeMillis();
 // directly hmset
 for (int i=0;i<10000;i++) {
  data.clear();
  data.put("k_" + i, "v_" + i);
  redis.hmset("key_" + i, data);
 }
 long end = System.currentTimeMillis();
 System.out.println("dbsize:[" + redis.dbSize() + "] .. ");
 System.out.println("hmset without pipeline used [" + (end - start) / 1000 + "] seconds ..");
 redis.select(8);
 redis.flushDB();
 // use pipeline hmset
 Pipeline p = redis.pipelined();
 start = System.currentTimeMillis();
 for (int i=0;i<10000;i++) {
  data.clear();
  data.put("k_" + i, "v_" + i);
  p.hmset("key_" + i, data);
 }
 p.sync();
 end = System.currentTimeMillis();
 System.out.println("dbsize:[" + redis.dbSize() + "] .. ");
 System.out.println("hmset with pipeline used [" + (end - start) / 1000 + "] seconds ..");
 
 //hmget 
 Set<String> keys = redis.keys("*");
 // Direct use of Jedis hgetall
 start = System.currentTimeMillis();
 Map<String,Map<String,String>> result = new HashMap<String,Map<String,String>>();
 for(String key : keys) {
  result.put(key, redis.hgetAll(key));
 }
 end = System.currentTimeMillis();
 System.out.println("result size:[" + result.size() + "] ..");
 System.out.println("hgetAll without pipeline used [" + (end - start) / 1000 + "] seconds ..");
 
 // use pipeline hgetall
 Map<String,Response<Map<String,String>>> responses = new HashMap<String,Response<Map<String,String>>>(keys.size());
 result.clear();
 start = System.currentTimeMillis();
 for(String key : keys) {
  responses.put(key, p.hgetAll(key));
 }
 p.sync();
 for(String k : responses.keySet()) {
  result.put(k, responses.get(k).get());
 }
 end = System.currentTimeMillis();
 System.out.println("result size:[" + result.size() + "] ..");
 System.out.println("hgetAll with pipeline used [" + (end - start) / 1000 + "] seconds ..");
 
 redis.disconnect();
 
 }
 
 
}

The test results are as follows:

dbsize:[10000] .. 
hmset without pipeline used [243] seconds .. 
dbsize:[10000] .. 
hmset with pipeline used [0] seconds .. 
result size:[10000] .. 
hgetAll without pipeline used [243] seconds .. 
result size:[10000] .. 
hgetAll with pipeline used [0] seconds .. 

Use pipeline to read and write 10,000 records in batches, that is, a small dish.