Summary analysis of Redis connection errors

  • 2020-06-19 12:02:12
  • OfStack

preface

Recently, redis experienced a series of errors due to increased traffic, such as:

LOADING Redis is loading the dataset in memory use of closed network connection connection pool exhausted connection refuse by peer

So let's analyze each of them.

LOADING Redis is loading the dataset in memory

There are at least two possibilities here

The available memory is too small, it can be solved by modifying maxmemory in ES17en. conf redis was loading the ES21en. rdb file at startup and was unavailable at startup due to slow loading

I've met is the 2nd kind of circumstance, AWS in automatic expansion, every new instances of EC2 error, reason is redis at startup found a dump. rdb, then go to load it, lead to the service in the server error, then withdrew, and redis load this for a long time (don't know why), supervisord automatic restart after a new service is still an error.

Later, the dump.rdb file in the image was deleted so that the service could start normally.

The dump.rdb file may have been generated because of an error in redis that was included in the mirroring process, causing each of the newly generated instances to report an error.

Lesson learned this time, stop drop redis and then delete ES46en.rdb before making the next image.

The other three types of mistakes

1 is also a variety of data at the beginning, and then a variety of configuration, resulting in the three errors have appeared.

1 At the beginning, I thought that the golang code did not handle the connection exception of redis correctly, so I upgraded redigo and changed the configuration of timeout, max_active, wait and so on in golang, but found it was of no use.

After about a week of this back and forth, I finally found something fishy in pool.Active and pool.MaxActive.

Because I set MaxActive as 10000, I opened 10,000 go runtine to test it, and found that the current connection number of pool.Active was always around 4000, and then all kinds of errors were reported.

At that time, I was also short-circuited in my mind. I always thought that redigo did not properly handle the connection of redis, which caused pool.Active could not reach the maximum. Always thinking about changing the code for redigo...

Later, I had no choice but to change 1 to ulimit, the old one was 500000, but changed to 990000. I found that the connection error was still reported, and ES88en. Active could not go up. Check online and sure enough, redis-ES92en has an maxclients configuration... The default is more than 4000, changed to 10000, the whole world is quiet......

I can't really blame you, because redigo also has an max_active parameter, who knows what the ES99en-ES100en parameter is?

Redis for configuration of highly concurrent services

Redis client (i.e. golang code)

Wait: true If the connection pool is full, wait. Redis handles very quickly
IdleTimeout: 5s 1 business logic 5s can't handle, then you should optimize your code. If it is set to 0, 10000, the connection will be lost and the server will not be able to retrieve it. A zombie connection will be generated.

MaxActive: 10000 equals 10000 concurrency per second for this server.

Redis server (es126EN-server)

maxclients should be larger than MaxActive

Additional question: How to calculate the maximum number of files on 1 server?

The number of people in the United States who have been in the United States for more than a decade has increased

[

this ends up being about 100 for every 1MB of ram.

]

For example, if 4G memory, the maximum number of open files can be set to: 4 * 1024 * 100 = 409600

conclusion


Related articles: