Redis optimization experience summary of must read

Memory management optimization

Redis Hash is an value with an HashMap inside. If the number of members of Map is relatively small, the Map will be stored in a compact format similar to the 1-dimensional linear format, that is, the memory overhead of a large number of Pointers will be saved. This parameter control corresponds to the following two items in the redis.conf configuration file:

hash-max-zipmap-entries 64 hash-max-zipmap-value 512

When value does not have more than a few members inside the Map, it will be stored in a linear compact format. The default is 64, that is, if value has less than 64 members inside, it will be stored in a linear compact format, and beyond this value, it will be automatically converted to the true HashMap.

hash-max-zipmap-value means that when value, each member value within Map is no more than a few bytes long, it is stored in linear compact form to save space.

Conditions for the above two conditions for any one more than the value will be converted into real HashMap, also won't save memory, then the value is not set the bigger the better, of course, the answer is no, HashMap advantage is to find and the time complexity of the operation are O (1), and give up Hash using 1 d storage is O (n) time complexity, if

If the number of members is small, the impact will be small; otherwise, the performance will be seriously affected. Therefore, it is necessary to balance the setting of this value, which is still the fundamental balance between time cost and space cost in general.

list-max-ziplist-value 64 list-max-ziplist-entries 512

The compact storage format will be used for the number of bytes less than the node value size of list data type, and the compact storage format will be used for the number of nodes of list data type below.

Memory pre-allocation:

Redis internal implementation is not to do too much memory allocation optimization (contrast Memcache), exist in 1 set extent memory fragments, but in most cases this will not become Redis performance bottlenecks, but if the Redis internal storage most of the data is numeric, Redis internal adopted a shared integer way to save memory overhead, namely when the system starts to assign 1 from 1 ~ n then multiple numerical objects placed in a pool, If the stored data is within the scope of this numerical data, then remove this object from the pool directly, and through the way of the reference count to share, so under the system to store a large amount of numerical, can also be 1 extent save memory and improve the performance of the parameter values n Settings need to modify the source code of 1 row REDIS_SHARED_INTEGERS macro definition, the default is 10000, can according to their own needs to be modified, the modified recompiling.

Persistence mechanism:

Timed snapshot method (snapshot) :

The persistent way is actually within the Redis a timer event, every time to check whether the number of changes in the current data and time to meet the configuration of persistent trigger conditions, if meet the fork via the operating system calls to create a child process, the child process will default to the parent process share the same address space, then you can through the child to traverse the entire memory to store operation, and the main process can still provide services, When there are writes, the operating system performs them in memory pages (page) to ensure that the parent and child processes do not interact with each other.

The main disadvantage of this persistence is that the timed snapshot only represents a memory image for a period of time, so a system reboot loses all the data between the last snapshot and the reboot.

Statement append method (aof) :

The aof mode is actually similar to the statement based binlog mode of mysql, that is, every command that changes the memory data of Redis will be appended to an log file, that is, this log file is the persistent data of Redis.

aof way may lead to additional log are the main disadvantages of file size is too large, when the system restart recovery data if is aof load data can be very slow, some 10 G data may take several hours to finish loading, and, of course, this time not for disk file read speed slow, but by reading all the commands to perform 1 times in memory. In addition, since log has to be written for every command, the read-write performance of Redis will also decrease if aof is used.

You can consider saving the data into different instances of Redis, each with a memory size of about 2G, to avoid putting your eggs in one basket. This can not only reduce the impact of cache failure on the system, but also speed up the data recovery. However, it also brings a certain amount of complexity to the system design.

Redis persistent crash issues:

Redis online operations experience will find Redis in physical memory use is more, but no more than the actual physical instability occurs when the amount of ram and even collapse of the problem, some people think that is based on the snapshot persistent fork cause memory footprint due to the double system calls, this view is not accurate, because fork call copy on - write mechanism is based on the operating system, pages of this unit is only those who have written the dirty pages can be copied, But 1 your system will not copy all the pages in a short time due to write in, so what causes Redis to crash?

Answer is Redis persistence USES the Buffer IO made a so-called Buffer IO refers to Redis for persistent file write and read operations will use physical memory Page Cache, most database systems will use Direct IO to bypass this Page Cache Cache and maintains a data, and when the Redis persistence file is too large (especially the snapshot file), And read and write on it, the data in disk files will be loaded into physical memory as an operating system for the file Cache 1 layer, the layer Cache data with Redis memory management is actually repeated storage of data, while the kernel in physical memory they do Page Cache out of work, but the kernel are likely to think that one piece of Page Cache is more important, your process start Swap, then your system will begin to appear unstable or collapsed. Our experience is that when your Redis physical memory USES more than 3/5 of its total memory capacity, it becomes dangerous.

Conclusion:

1. Select appropriate data types according to business needs, and set corresponding compact storage parameters for different application scenarios.

2. When business scenarios do not require data persistence, turning off all persistence methods can achieve the best performance and maximum memory usage.

3. If persistence is needed, select 1 between snapshot mode and statement append mode according to whether it is acceptable to restart some lost data. Do not use virtual memory and diskstore mode.

4. Don't allow your Redis machine to use more than 3/5 of its physical memory.

The maxmemory option in redis.conf tells Redis to reject subsequent write requests after using too much physical memory. This parameter can protect your Redis from using too much physical memory and causing swap to crash.

vm-enabled is no in redis.conf