Detail the data expiration policy in Redis

1. The expiration time of key in Redis

Set the expiration time of the data through the EXPIRE key seconds command. A return of 1 indicates that the setting was successful, and a return of 0 indicates that key does not exist or fails to set the expiration time successfully. When the expiration time is set on key, key will be automatically deleted after a specified number of seconds. key with a specified expiration time is said to be unstable in Redis.

The expiration time associated with key is cleared when it is deleted by the DEL command or reset by the SET or GETSET commands

127.0.0.1:6379> setex s 20 1
OK
127.0.0.1:6379> ttl s
(integer) 17
127.0.0.1:6379> setex s 200 1
OK
127.0.0.1:6379> ttl s
(integer) 195
127.0.0.1:6379> setrange s 3 100
(integer) 6
127.0.0.1:6379> ttl s
(integer) 152
127.0.0.1:6379> get s
"1\x00\x00100"
127.0.0.1:6379> ttl s
(integer) 108
127.0.0.1:6379> getset s 200
"1\x00\x00100"
127.0.0.1:6379> get s
"200"
127.0.0.1:6379> ttl s
(integer) -1

Use PERSIST to clear the expiration time

127.0.0.1:6379> setex s 100 test
OK
127.0.0.1:6379> get s
"test"
127.0.0.1:6379> ttl s
(integer) 94
127.0.0.1:6379> type s
string
127.0.0.1:6379> strlen s
(integer) 4
127.0.0.1:6379> persist s
(integer) 1
127.0.0.1:6379> ttl s
(integer) -1
127.0.0.1:6379> get s
"test"

Using rename only changes the key value

127.0.0.1:6379> expire s 200
(integer) 1
127.0.0.1:6379> ttl s
(integer) 198
127.0.0.1:6379> rename s ss
OK
127.0.0.1:6379> ttl ss
(integer) 187
127.0.0.1:6379> type ss
string
127.0.0.1:6379> get ss
"test"

Note: after Redis2.6, the precision of expire can be controlled within 0 to 1 millisecond. The expired information of key is stored in the form of absolute Unix timestamp (after Redis2.6, it is stored in millisecond precision). Therefore, when multiple servers are synchronized, 1 must synchronize the time of each server

2. Redis expiration key deletion policy

There are three ways for Redis key to expire:

Passive deletion: when reading/writing an expired key, the lazy deletion policy is triggered to delete the expired key directly Active deletion: since the lazy deletion policy cannot guarantee the timely deletion of cold data, Redis will regularly and actively eliminate 1 batch of expired key The active cleanup policy is triggered when the current used memory exceeds the maxmemory limit

Passive delete

Only when key is operated on (e.g., GET) will REDIS passively check whether the key is expired, and if so, delete it and return NIL.

1. This deletion strategy is friendly to CPU. The deletion operation will only be carried out if necessary, and no unnecessary CPU time will be wasted on other expire key.

2. However, this strategy is not memory friendly. One key has expired, but it will not be deleted before it is operated, and still occupies memory space. If a large number of expiration keys exist but are rarely accessed, a large amount of memory space will be wasted. expireIfNeeded(redisDb *db, robj *key) The function is located at src/ db.c.

/*-----------------------------------------------------------------------------
 * Expires API
 *----------------------------------------------------------------------------*/
 
int removeExpire(redisDb *db, robj *key) {
 /* An expire may only be removed if there is a corresponding entry in the
 * main dict. Otherwise, the key will never be freed. */
 redisAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) != NULL);
 return dictDelete(db->expires,key->ptr) == DICT_OK;
}
 
void setExpire(redisDb *db, robj *key, long long when) {
 dictEntry *kde, *de;
 
 /* Reuse the sds from the main dict in the expire dict */
 kde = dictFind(db->dict,key->ptr);
 redisAssertWithInfo(NULL,key,kde != NULL);
 de = dictReplaceRaw(db->expires,dictGetKey(kde));
 dictSetSignedIntegerVal(de,when);
}
 
/* Return the expire time of the specified key, or -1 if no expire
 * is associated with this key (i.e. the key is non volatile) */
long long getExpire(redisDb *db, robj *key) {
 dictEntry *de;
 
 /* No expire? return ASAP */
 if (dictSize(db->expires) == 0 ||
 (de = dictFind(db->expires,key->ptr)) == NULL) return -1;
 
 /* The entry was found in the expire dict, this means it should also
 * be present in the main dict (safety check). */
 redisAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) != NULL);
 return dictGetSignedIntegerVal(de);
}
 
/* Propagate expires into slaves and the AOF file.
 * When a key expires in the master, a DEL operation for this key is sent
 * to all the slaves and the AOF file if enabled.
 *
 * This way the key expiry is centralized in one place, and since both
 * AOF and the master->slave link guarantee operation ordering, everything
 * will be consistent even if we allow write operations against expiring
 * keys. */
void propagateExpire(redisDb *db, robj *key) {
 robj *argv[2];
 
 argv[0] = shared.del;
 argv[1] = key;
 incrRefCount(argv[0]);
 incrRefCount(argv[1]);
 
 if (server.aof_state != REDIS_AOF_OFF)
 feedAppendOnlyFile(server.delCommand,db->id,argv,2);
 replicationFeedSlaves(server.slaves,db->id,argv,2);
 
 decrRefCount(argv[0]);
 decrRefCount(argv[1]);
}
 
int expireIfNeeded(redisDb *db, robj *key) {
 mstime_t when = getExpire(db,key);
 mstime_t now;
 
 if (when < 0) return 0; /* No expire for this key */ /* Don't expire anything while loading. It will be done later. */ if (server.loading) return 0; /* If we are in the context of a Lua script, we claim that time is * blocked to when the Lua script started. This way a key can expire * only the first time it is accessed and not in the middle of the * script execution, making propagation to slaves / AOF consistent. * See issue #1525 on Github for more information. */ now = server.lua_caller ? server.lua_time_start : mstime(); /* If we are running in the context of a slave, return ASAP: * the slave key expiration is controlled by the master that will * send us synthesized DEL operations for expired keys. * * Still we try to return the right information to the caller, * that is, 0 if we think the key should be still valid, 1 if * we think the key is expired at this time. */ if (server.masterhost != NULL) return now > when;
 
 /* Return when this key has not expired */
 if (now <= when) return 0; /* Delete the key */ server.stat_expiredkeys++; propagateExpire(db,key); notifyKeyspaceEvent(REDIS_NOTIFY_EXPIRED, "expired",key,db->id);
 return dbDelete(db,key);
}
 
/*-----------------------------------------------------------------------------
 * Expires Commands
 *----------------------------------------------------------------------------*/
 
/* This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT
 * and PEXPIREAT. Because the commad second argument may be relative or absolute
 * the "basetime" argument is used to signal what the base time is (either 0
 * for *AT variants of the command, or the current time for relative expires).
 *
 * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for
 * the argv[2] parameter. The basetime is always specified in milliseconds. */
void expireGenericCommand(redisClient *c, long long basetime, int unit) {
 robj *key = c->argv[1], *param = c->argv[2];
 long long when; /* unix time in milliseconds when the key will expire. */
 
 if (getLongLongFromObjectOrReply(c, param, &when, NULL) != REDIS_OK)
 return;
 
 if (unit == UNIT_SECONDS) when *= 1000;
 when += basetime;
 
 /* No key, return zero. */
 if (lookupKeyRead(c->db,key) == NULL) {
 addReply(c,shared.czero);
 return;
 }
 
 /* EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past
 * should never be executed as a DEL when load the AOF or in the context
 * of a slave instance.
 *
 * Instead we take the other branch of the IF statement setting an expire
 * (possibly in the past) and wait for an explicit DEL from the master. */
 if (when <= mstime() && !server.loading && !server.masterhost) { robj *aux; redisAssertWithInfo(c,key,dbDelete(c->db,key));
 server.dirty++;
 
 /* Replicate/AOF this as an explicit DEL. */
 aux = createStringObject("DEL",3);
 rewriteClientCommandVector(c,2,aux,key);
 decrRefCount(aux);
 signalModifiedKey(c->db,key);
 notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"del",key,c->db->id);
 addReply(c, shared.cone);
 return;
 } else {
 setExpire(c->db,key,when);
 addReply(c,shared.cone);
 signalModifiedKey(c->db,key);
 notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"expire",key,c->db->id);
 server.dirty++;
 return;
 }
}
 
void expireCommand(redisClient *c) {
 expireGenericCommand(c,mstime(),UNIT_SECONDS);
}
 
void expireatCommand(redisClient *c) {
 expireGenericCommand(c,0,UNIT_SECONDS);
}
 
void pexpireCommand(redisClient *c) {
 expireGenericCommand(c,mstime(),UNIT_MILLISECONDS);
}
 
void pexpireatCommand(redisClient *c) {
 expireGenericCommand(c,0,UNIT_MILLISECONDS);
}
 
void ttlGenericCommand(redisClient *c, int output_ms) {
 long long expire, ttl = -1;
 
 /* If the key does not exist at all, return -2 */
 if (lookupKeyRead(c->db,c->argv[1]) == NULL) {
 addReplyLongLong(c,-2);
 return;
 }
 /* The key exists. Return -1 if it has no expire, or the actual
 * TTL value otherwise. */
 expire = getExpire(c->db,c->argv[1]);
 if (expire != -1) {
 ttl = expire-mstime();
 if (ttl < 0) ttl = 0; } if (ttl == -1) { addReplyLongLong(c,-1); } else { addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000)); } } void ttlCommand(redisClient *c) { ttlGenericCommand(c, 0); } void pttlCommand(redisClient *c) { ttlGenericCommand(c, 1); } void persistCommand(redisClient *c) { dictEntry *de; de = dictFind(c->db->dict,c->argv[1]->ptr);
 if (de == NULL) {
 addReply(c,shared.czero);
 } else {
 if (removeExpire(c->db,c->argv[1])) {
  addReply(c,shared.cone);
  server.dirty++;
 } else {
  addReply(c,shared.czero);
 }
 }
}

But this is not enough, because there may be some key 1 will never be again to access, these Settings the expiration time key also need to be removed after the expiration, we can even see it as a kind of memory leaks, useless garbage data takes up a lot of memory, and the server is not to release them, this very rely on memory for running state Redis server, be sure it is not a good news

Take the initiative to delete

First, let's talk about 1 time event. For a continuously running server, the server needs to check and sort out its own resources and state on a regular basis, so as to keep the server in a healthy and stable state. Such operations are collectively referred to as general operations (cron job).

In Redis, regular operations are implemented by redis.c /serverCron, which does the following

Update server statistics such as time, memory footprint, database footprint, etc. Clean up expired key-value pairs in the database. Resize an unreasonable database. Close and clean failed clients. Try AOF or RDB persistence. If the server is the primary node, periodically synchronize the secondary nodes. If in cluster mode, periodically synchronize and connect to the cluster.

Redis runs serverCron as a time event to ensure that it is automatically run once every 1 interval, and because serverCron needs to run 1 directly during Redis's server run, it is a cyclic time event: serverCron runs 1 directly until the server is down.

In version 2.6 of Redis, the program specifies that serverCron will run 10 times per second, on average once every 100 milliseconds. Starting with Redis 2.8, users can adjust the number of times per second of serverCron by modifying the hz option. For details, please refer to the redis.conf file for the hz option

Also known as timed deletion, "scheduled" here refers to the cleanup policy that is triggered by Redis on a regular basis, which is located at src/ redis.c activeExpireCycle(void) Function to do that.

serverCron is a positioning task driven by the event framework of redis. In this timing task, activeExpireCycle function will be called. For each db, REDIS_EXPIRELOOKUPS_TIME_LIMIT, the expired key may be deleted much later within the limited time. This active deletion strategy makes up for the memory unfriendliness of the passive deletion strategy.

Therefore, Redis will periodically randomly test and process 1 batch of key with expiration time set. The expired key tested will be deleted.

Typically,Redis does the following 10 times per second:

Randomly test 100 key with expiration time set Delete all found expired key Repeat step 1 if more than 25 key are deleted

This is a simple algorithm based on probability. The basic assumption is that the sample extracted can represent the entire key space, and redis keeps cleaning the expired data until the percentage of key that is about to expire falls below 25%. This also means that the amount of key that is expired but still occupies memory space at any given time is, at most, the number of writes per second divided by 4.

The default value in Redis-3.0.0 is 10, representing 10 background tasks invoked per second.

In addition to the frequency of active elimination, Redis also has a limit on the maximum time to execute each elimination task, so as to ensure that each active elimination will not block application requests too much. The formula for calculating this limit is as follows:

#define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 /* CPU max % for keys collection */ 
... 
timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;

Increasing the size of hz will increase the frequency of Redis's active obsoletion. If your Redis store contains a lot of cold data and is taking up too much memory, consider increasing this value, but the authors of Redis recommend not exceeding 100. We actually increased this value to 100 on the line and observed a 2% increase in CPU, but there was a significant increase in the memory release rate for cold data (by looking at the number of keyspace and the size of used_memory).

It can be seen that timelimit and server.hz have a reciprocal relationship, which means that the larger the hz configuration is, the smaller the timelimit will be. In other words, the higher the expected frequency of active elimination per second, the shorter the maximum elapsed time per elimination. The maximum culling time per second is fixed at 250ms (1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/100), while the frequency of culling and the maximum time per culling are controlled by the hz parameter.

From the above analysis, when the expired key ratio in redis does not exceed 25%, increasing hz can significantly increase the minimum number of scanned key. If hz is 10, scan at least 200 key in 1 second (call 10 times in 1 second * randomly extract at least 20 key each time). If hz is changed to 100, scan at least 2,000 key in 1 second; On the other hand, if the expired key ratio exceeds 25%, there is no upper limit to the number of key to be scanned, but the cpu time takes up a maximum of 250ms per second.

When REDIS is running in master-slave mode, only the master node will execute the above two expiration deletion strategies, and then the deletion operation "del key" will be synchronized to the slave node.

maxmemory

Active cleanup policy is triggered when currently used memory exceeds the maxmemory limit

volatile-lru: LRU only for key with expiration time set (default) allkeys-lru: delete key of the lru algorithm volatile-random: random deletion is about to expire key allkeys-random: random deletion volatile-ttl: delete that which is about to expire noeviction: never expires, returns error when mem_used memory has exceeded the setting of maxmemory, the redis.c /freeMemoryIfNeeded(void) function is triggered for all read and write requests to clean up the excess memory. Note that the cleanup process is blocked until enough memory space is cleared. So if maxmemory is reached and the caller is still writing, the active cleanup strategy may be triggered repeatedly, resulting in a definite delay in the request.

When mem_used memory exceeds the maxmemory setting, all read and write requests are triggered redis.c/freeMemoryIfNeeded(void) Function to clean up excess memory. Note that the cleanup process is blocked until enough memory space is cleared. So if maxmemory is reached and the caller is still writing, the active cleanup strategy may be triggered repeatedly, resulting in a definite delay in the request.

When cleaning up, appropriate cleaning will be done according to the maxmemory-policy configured by the user (1 is LRU or TTL generally). The LRU or TTL policy here is not for all key of redis, but for the sample pool of maxmemory-samples in the configuration file.

The default configuration of maxmemory-samples in redis-3.0.0 is 5, which, if added, will improve the accuracy of LRU or TTL. The author of redis has tested that when this configuration is 10, it is very close to the accuracy of full LRU, and the increase of maxmemory-samples will lead to the consumption of more CPU time in active cleaning. It is recommended that:

Try not to trigger maxmemory, preferably after the memory footprint of mem_used has reached the 1 set ratio of maxmemory, you should consider increasing hz to speed up the phasing out, or cluster expansion. If you can control the memory, you don't need to modify the maxmemory-samples configuration. If Redis itself is LRU cache service (this service 1 is in maxmemory state for a long time, and Redis automatically makes LRU obsolete), maxmemory-samples can be appropriately increased.

The following is a description of the configuration parameters mentioned above

# Redis calls an internal function to perform many background tasks, like 
# closing connections of clients in timeout, purging expired keys that are 
# never requested, and so forth. 
# 
# Not all tasks are performed with the same frequency, but Redis checks for 
# tasks to perform according to the specified "hz" value. 
# 
# By default "hz" is set to 10. Raising the value will use more CPU when 
# Redis is idle, but at the same time will make Redis more responsive when 
# there are many keys expiring at the same time, and timeouts may be 
# handled with more precision. 
# 
# The range is between 1 and 500, however a value over 100 is usually not 
# a good idea. Most users should use the default of 10 and raise this up to 
# 100 only in environments where very low latency is required. 
hz 10 
 
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory 
# is reached. You can select among five behaviors: 
# 
# volatile-lru -> remove the key with an expire set using an LRU algorithm 
# allkeys-lru -> remove any key according to the LRU algorithm 
# volatile-random -> remove a random key with an expire set 
# allkeys-random -> remove a random key, any key 
# volatile-ttl -> remove the key with the nearest expire time (minor TTL) 
# noeviction -> don't expire at all, just return an error on write operations 
# 
# Note: with any of the above policies, Redis will return an error on write 
# operations, when there are no suitable keys for eviction. 
# 
# At the date of writing these commands are: set setnx setex append 
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd 
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby 
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby 
# getset mset msetnx exec sort 
# 
# The default is: 
# 
maxmemory-policy noeviction 
 
# LRU and minimal TTL algorithms are not precise algorithms but approximated 
# algorithms (in order to save memory), so you can tune it for speed or 
# accuracy. For default Redis will check five keys and pick the one that was 
# used less recently, you can change the sample size using the following 
# configuration directive. 
# 
# The default of 5 produces good enough results. 10 Approximates very closely 
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate. 
# 
maxmemory-samples 5

Expiration processing in Replication link and AOF files

In order to get the right behavior without causing 1 problem, when an key expires, the DEL operation will be recorded in the AOF file and passed to all the relevant slave. In other words, the expiration delete system 1 is carried out in the master instance and passed down, instead of being controlled by each salve. So you don't have a situation where the data doesn't add up to 1. When slave is connected to master, the expired key cannot be cleaned immediately (it needs to wait for the DEL operation passed by master). slave still needs to manage and maintain the expired state in the data set so that when slave is promoted to master, it can be expired independently like master1.

conclusion

The above is the whole content of this article, I hope the content of this article to your study or work can bring 1 definite help, if you have questions you can leave a message to communicate.