Redis gets a script instance of a large key value

1, the preface

In work, there are often instances of Redis that are quite large due to inappropriate use, inaccurate business estimates, or not timely processing of key.

So what are the problems with large Key?

If it is in cluster mode, load balancing cannot be done, which causes the request to be tilted to an instance, and the QPS of this instance will be relatively large and the memory consumption will be large. For Redis single-thread model, CPU bottleneck is easy to occur. When there is a bottleneck in memory, only vertical storage capacity can be used, and more excellent servers can be used.

Involves the operation of the large key, especially using hgetall, lrange 1-0, get, hmget operation, such as network card may be a bottleneck, also to other operation, cause jams qps is likely, a sudden drop or sudden rise in the number of trend looks 10 points is not smooth, serious will cause the application to connect, instance or cluster in certain time period, not with the state.

If the key needs to be deleted, if the DEL operation is carried out directly, the operated instance will be occupied by Block, resulting in the failure to respond to the application request, and the time of the Block will be longer as the key gets larger.

In redis, for a very large key, such as the hash type, it would be very slow to look at the value directly, so I thought of writing a script to get it by incremental iteration. I won't say much more, but let's start with 1 and look at the details.

2. The specific script is as follows:

Function: Scan redis for all elements in an key

Usage: python bigkey_save_values.py "zyyset" "*" 100

3. python script is implemented as follows:

#!/usr/bin/env python
# -*- coding: UTF-8 -*- 
# Function: Scan a key All of the elements in it 
# Usage: python bigkey_save_values.py "zyyset" "m*" 100
# To get all elements: python bigkey_save_values.py "zyyset" "*" 100
__author__ = "lcl" 
import sys
import redis 
import os 
pool=redis.ConnectionPool(host='192.168.225.128',port=6379,db=0) 
r = redis.StrictRedis(connection_pool=pool) 
# The specified key
key = sys.argv[1]
# Scan matching value 
match = sys.argv[2]
# Number of matches per time 
count = sys.argv[3]
# The total number 
total = 0
# Get current path 
path = os.getcwd()
keytype = r.type(key)
print "key The type of %s" % (keytype)
# Scan the key Output file 
txt = path+"/%s.txt" % (key)
# Open the file 
#f = open(txt,"w")
def list_iter(name):
 list_count = r.llen(key)
 for index in range(list_count):
  yield r.lindex(key, index)
if keytype=='list':
 f = open(txt,"w")
 for item in list_iter(key):
# Matches to the corresponding key In the member/score Output to a file 
  f.write("%s %s" % (item,"\n"))
  total = total+1
 f.close
elif keytype=='hash':
 f = open(txt,"w")
 for item in r.hscan_iter(key,match = match,count = count):
# Matches to the corresponding key In the member/score Output to a file 
  f.write("%s %s" % (item,"\n"))
  total = total+1
 f.close
elif keytype=='set':
 f = open(txt,"w")
 for item in r.sscan_iter(key,match = match,count = count):
  f.write("%s %s" % (item,"\n"))
  total = total+1
 f.close
elif keytype=='zset':
 f = open(txt,"w")
 for item in r.zscan_iter(key,match = match,count = count):
  f.write("%s %s" % (item,"\n"))
  total = total+1
 f.close
else:
 print("key The type of string . value To: " + r.get(key))
print "key:%s the match : %s Is: %d" % (key,match,total)

conclusion