redis's bigkey scan script is introduced in depth

preface

It is well known that the existence of large key in redis is a very dangerous thing. As recent work has shifted to middleware related work, the bigkey scanning approach has been looked at in 1. Firstly, the scanning script provided by Aliyun is introduced:

Specific visible: https: / / yq aliyun. com/articles / 117042 & # 63; t = t1

I ran a stress test on this script, and the memory of redis was 15G, the number of key was 2KW, and the number of ops was 40K to 80K. In this case, the Script of Aliyun could not run successfully at all (estimated time in days), mainly because it needed to interact with redis many times for each confirmation of key. Therefore, I modified the script to include the pipeline and debug object methods. The script is as follows:

import sys
import redis
 
 
def find_big_key_normal(db_host, db_port, db_password, db_num):
 client = redis.StrictRedis(host=db_host, port=db_port, password=db_password, db=db_num)
 i=0
 temp = client.scan(cursor=i,count=1000)
 j =0
 while temp[0]>0 :
 i=temp[0]
 j=j+len(temp[1])
 try:
  r = client.pipeline(transaction=False)
  for k in temp[1]:
  r.debug_object(k)
  tempA = r.execute()
  x = 0
  for key in tempA:
  length = key.get("serializedlength")
  ##type = key.get("encoding")
  if length > 10240 :
   type = client.type(temp[1][x])
   print temp[1][x], type,length
  x=x+1
 except :
  print "a execption come"
 temp = client.scan(cursor=i,count=1000)
 
 
if __name__ == '__main__':
 if len(sys.argv) != 4:
  print 'Usage: python ', sys.argv[0], ' host port password '
  exit(1)
 db_host = sys.argv[1]
 db_port = sys.argv[2]
 db_password = sys.argv[3]
 r = redis.StrictRedis(host=db_host, port=int(db_port), password=db_password)
 nodecount = 1
 keyspace_info = r.info("keyspace")
 for db in keyspace_info:
 print 'check ', db, ' ', keyspace_info[db]
 find_big_key_normal(db_host, db_port, db_password, db.replace("db", ""))

I also ran a stress test on the above script, with the memory of redis between 15G, the number of key between 2KW, and the number of ops between 40K and 80K. In this case:

The script runs in 10 minutes and is fully available.

Note: The script of Aliyun supports cluster. My script only supports stand-alone computers. If you are interested, you can change it by yourself.

conclusion