Performance problems with redis's hGetAll function of remember Redis's cheating HGETALL

Before didn't pay attention to the function, 1 straight with Memcache way of data storage, but since replaced redis, for 1 hash data save and take convenient for Memcache very much, but the problem is coming, a hash list if the quantity is small, almost can't see the problem, using hGetAll function as the the list of more than 50 or more, this time with hGetAll function can be very intuitive see performance issues, there is not a data analysis.

Redis is single-threaded! While it processes one request, the rest of the requests have to wait. Requests are usually processed quickly, but when we used HGETALL, we had to traverse each field to get the data, and the amount of CPU resources consumed was proportional to the number of fields, which would be even worse if we also used PIPELINING.

PERFORMANCE = CPUs / OPERATIONs

In other words, in order to improve performance in this scenario, either increase the number of CPU in the calculation process; Or reduce the number of operations in the process. In order to continue to use the hash data structure, and to solve this problem, a more convenient way is to store hash as a serialized string, and then extract the deserialized data before using hGet(key,array(hash..). ).

Such as:

....

$arrKey = array('dbfba184bef630526a75f2cd073a6098','dbfba184bef630526a75f2cd0dswet98')

$strKey = 'test';

$obj->hGet($strKey,$arrKey);

The original hGetAll operation is simplified to hGet, that is, it is no longer necessary to traverse every field in hash. Therefore, even if multiple CPU cannot be involved in the operation, the number of operations is greatly reduced, so the performance improvement is still significant. Of course, the disadvantage is also obvious. Like all the redundant methods, this scheme wastes a lot of memory.

One might ask, how can this improve performance without the process of traversing a field, but without the process of deserializing it, which is often costly? The key to the problem is that the operations we initially performed to traverse the fields were performed on one cpu, and the subsequent deserialization operations, regardless of the language, can be guaranteed to be performed on multiple cpu by multiple processes or multiple threads, so the overall performance is improved.

In addition, many people intuitively solve problems by running multiple instances of redis. Indeed, this will increase the operation process of CPU quantity, helps to improve performance, but it is important to note that hGetAll and PIPELINING tend to bring out the number of operations in the process of operation level geometric explosive growth, by contrast, we can increase the quantity more redis instance is a drop in the bucket, so in this case, this method can not completely solve the problem.

Remember Redis the deceiver HGETALL

There is no pit in the world, but when many fall, they become pits.

I've heard people say that Redis's HGETALL is a pit, but I just don't believe in evil: no matter what pit, 1 must stomp on it before I give up. To say that this is less than the Yellow River heart die, say bad is not to see the coffin not to cry.

The program started out so stable that I wanted to send anyone who said HGETALL was a pit 1 word: yuck! At this time, I was like a frog in warm water and forgot the existence of danger. After one day and one day, I suddenly had to change the demand for one day. I had to expand the content of HASH data from 10 fields to more than 100 fields, and at the same time, I used Pipelining1 to obtain hundreds of HGETALL results. So I fell into a hole: the server went down.

Why is that? Redis is single-threaded! While it processes one request, the rest of the requests have to wait. Requests are usually processed quickly, but when we use HGETALL, we have to traverse each field to get the data, and the amount of CPU resources consumed is proportional to the number of fields. If we also use PIPELINING, it will be even worse.

How to solve this problem? Please allow me to give you a formula:

PERFORMANCE = CPUs / OPERATIONs

That is to say, in order to improve performance in this scenario, either increase the number of CPU in the calculation process; Or reduce the number of operations in the process. To be specific, I generally think of the following methods:

With the help of Memcached

Redis storage way not to make any changes, additional, we use Memcached 1 set of implement caching, stored inside that need to be in Redis HGETALL HASH, of course, as a result of the stored in Memcached are strings, so when we store HASH actually stored HASH serialized string, deserialization again can query, Memcached client driver can usually transparent realization of serialization and deserialization. The advantage of this scheme is that because Memcached supports multi-threading, more CPU can be involved in the operation. At the same time, the corresponding operation will be reduced as there is no need to traverse each field again. Of course, there are many disadvantages, because a new cache layer is introduced, so the memory is wasted, increasing the complexity. In addition, sometimes even if we only need to get a few fields of data, we have to query the complete data first, and then filter, which undoubtedly wastes the bandwidth. Of course we can query Redis directly in this case, but it certainly adds a little more complexity.

By the way, Memcached supports Multiget and can achieve a similar effect to Pipelining, but you should be especially careful about the pit of Memcached, also known as Mulitiget bottomless pit.

Serialized field redundancy

When storing HASH, Redis will save one more field named "all", whose content is the serialization of the original HASH data. In the actual query, only the redundant field of HGET will be deserialized. The advantage of this scheme is that we can reduce the original HGETALL operation to HGET by serializing redundant fields. That is to say, it is no longer necessary to traverse every field in HASH. Therefore, even if multiple CPU cannot be involved in the operation, the number of operations is greatly reduced, so the performance improvement is still significant. Of course, the disadvantage is also obvious. Like all the redundant methods, this scheme wastes a lot of memory.

One might ask, how can this improve performance without the process of traversing a field, but without the process of deserializing it, which is often costly? The key to the problem is that the first operations we performed to traverse the fields were performed on one CPU, and the subsequent deserialization operations, regardless of the language, can be guaranteed to be performed on multiple CPU by multiple processes or multiple threads, so the overall performance is improved.

...

In addition, many people's intuition is to solve the problem by running multiple instances of Redis. Indeed, this will increase the operation process of CPU quantity, helps to improve performance, but it is important to note that HGETALL and PIPELINING tend to bring out the number of operations in the process of operation level geometric explosive growth, by contrast, we can increase the quantity more Redis instance is a drop in the bucket, so in this case, this method can not completely solve the problem.

...

The pit is for stepping on. Don't be afraid to fall in, of course, if you can climb out!