Implement the SQL scaling method with Redis

This article mainly introduces the method of using Redis to achieve SQL scaling, including the aspects of locking and time series to improve the performance of traditional database. You can refer to it if you need.

Ease line competition

The early version we developed in Sentry was sentry.buffers. This is a simple system that allows us to implement very efficient buffering counters with the simple Last Write Wins policy. Importantly, we completely eliminated any form of durability with it (this is a very acceptable way for Sentry to work).

The operation is very simple. Every time an update comes in, we do the following steps:

Create a hash key bound to the incoming entity (hash key) Use HINCRBY to increment the counter value HSET all LWW data (e.g. "last seen ") Use the current timestamp ZADD hash key (hash key) to 1 "suspend" set

Now every time scale (10 seconds in Sentry) we dump (dump) these buffers and fan them out (fanout the writes). It looks like this:

Get all the key using ZRANGE Initiate 1 job to RabbitMQ for each suspended key

Now the RabbitMQ job will be able to read and clear the hash table, and the "pending" update has popped up 1 set. A few things to note:

In the following example we want to pop up the number of Settings with only 1 set we will use 1 set of sorts (for example we need the 100 old sets). If we end up doing multiple sorts in order to process 1 key value, this person will get no-oped due to another existing processing and hash clearing process. The system can scale over many Redis nodes simply by placing a 'mount' primary key on each node.

Once we had this model for handling the problem, we were able to ensure that "most of the time" only 1 row at a time in SQL would be updated immediately, which mitigated the locking problems we could have anticipated. This strategy is useful for Sentry, considering that you will be dealing with a scenario in which one data is suddenly generated and all the final combinations enter the same counter at one.

The speed limit

Because of sentry's limitations, we must end the continuing denial of service attacks. We address this by limiting connection speeds, one of which is supported by Redis. This is undoubtedly a more direct implementation within the scope of sentry.quotas.

The logic is straightforward, as shown below:

def incr_and_check_limit(user_id, limit): 
 key = '{user_id}:{epoch}'.format(user_id, int(time() / 60)) 
  
 pipe = redis.pipeline() 
 pipe.incr(key) 
 pipe.expire(key, 60) 
 current_rate, _ = pipe.execute() 
  
 return int(current_rate) > limit 

The method we have illustrated for limiting the rate is one of the most basic features of Redis on caching services: adding empty keys. Implementing the same behavior in a caching service might end up using this approach:

def incr_and_check_limit_memcache(user_id, limit): 
 key = '{user_id}:{epoch}'.format(user_id, int(time() / 60)) 
  
 if cache.add(key, 0, 60): 
  return False 
  
 current_rate = cache.incr(key) 
  
 return current_rate > limit 

In fact, we ended up with a strategy that allowed the sentinels to track short-term data on different events. In this case, we usually sort the user data so that we can find the data of the most active users in the shortest possible time.

The basic lock

Although Redis's is not highly available, our use case locks make it a good tool for working. We didn't use these at the core of the sentinel anymore, but a sample use case is that we want to minimize concurrency and simple no-action operations if things seem to be already running. This is useful for tasks that might need to be performed at intervals similar to cron, but without strong coordination.

Using SETNX in Redis like this is fairly simple:

from contextlib import contextmanagerr = Redis()@contextmanagerdef lock(key, nowait=True): 
 while not r.setnx(key, '1'): 
  if nowait: 
   raise Locked('try again soon!') 
  sleep(0.01) 
  
 # limit lock time to 10 seconds 
 r.expire(key, 10) 
  
 # do something crazy 
 yield 
  
 # explicitly unlock 
 r.delete(key) 

While the lock () inside the sentry takes advantage of the memcached, there is absolutely no reason we can't switch to Redis in it.
Time series data

Recently we created a new mechanism to store time series data in Sentry(included in sentry.tsdb). This is inspired by the RRD model, and Graphite in particular. We expect a quick and easy way to store short-term (say, 1-month) time series Numbers to handle high-speed write data, especially in extreme cases to calculate potential short-term rates. Although this is the first model, we still expect to store data in Redis, which is also a simple example of using counters.

In the current model, we use hash map of single 1 to store the entire time series data. For example, this means that all data items will have a data type and a one-second lifetime with a hash key. As follows:

{ 
 
  "<type enum>:<epoch>:<shard number>": { 
 
    "<id>": <count> 
 
  }} 

So in this case, we need to track the number of events. The event type maps to the enumeration type "1". The time for this determination is 1s, so our processing time needs to be in seconds. The hash ends up looking like this:

 { 
 
  "1:1399958363:0": { 
 
    "1": 53, 
 
    "2": 72, 
 
  }} 

A modifiable model may only use simple keys and add only one incremental register to the storage area.

"1:1399958363:0:1": 53 

We chose the hash mapping model for the following two reasons:

We can set all the keys to be 1 (this may also have a negative effect, but so far it's stable)

Compress the key value substantially, which is quite important

In addition, discrete numeric keys allow us to map virtual discrete key values to a fixed number of key values and assign a single 1 storage area (we can use 64, mapped to 32 physical nodes)

The data query is now complete by using Nydus and its map()(which depends on one workspace)(). The code for this operation is fairly robust, but fortunately it's not huge.

def get_range(self, model, keys, start, end, rollup=None): 
 """ To get a range of data for group ID=[1, 2, 3]: Start and end are both inclusive. >>> now = timezone.now() >>> get_keys(tsdb.models.group, [1, 2, 3], >>>   start=now - timedelta(days=1), >>>   end=now) """ 
 normalize_to_epoch = self.normalize_to_epoch 
 normalize_to_rollup = self.normalize_to_rollup 
 make_key = self.make_key 
  
 if rollup is None: 
  rollup = self.get_optimal_rollup(start, end) 
  
 results = [] 
 timestamp = end 
 with self.conn.map() as conn: 
  while timestamp >= start: 
   real_epoch = normalize_to_epoch(timestamp, rollup) 
   norm_epoch = normalize_to_rollup(timestamp, rollup) 
  
   for key in keys: 
    model_key = self.get_model_key(key) 
    hash_key = make_key(model, norm_epoch, model_key) 
    results.append((real_epoch, key, conn.hget(hash_key, model_key))) 
  
   timestamp = timestamp - timedelta(seconds=rollup) 
  
 results_by_key = defaultdict(dict) 
 for epoch, key, count in results: 
  results_by_key[key][epoch] = int(count or 0) 
  
 for key, points in results_by_key.iteritems(): 
  results_by_key[key] = sorted(points.items()) 
 return dict(results_by_key) 

It comes down to this:

Generate the necessary keys. Using the workspace, extract the minimum result set for all join operations (Nydus is responsible for these). Gives the results and maps them to the current storage area based on the specified time interval and the given key value.

The above is how to use Redis to achieve SQL scaling method, I hope to help you learn.