python implements redis three cas transaction operations

  • 2020-06-19 10:39:57
  • OfStack

cas, compare and set, is a typical transaction operation.

Simply put, a transaction is to access the same 1 data in the database without breaking the isolation and atomicity of the operation, thus ensuring the 1 data.

How do general databases, such as MySql, ensure the consistency of data 1? Locking, pessimistic locking. For example, when accessing a piece of data in the database, SELECT FOR UPDATE is used, and MySql locks the data until the transaction is committed (COMMIT) or rolled back (ROLLBACK). If at this point another transaction writes to the locked data, the transaction will be blocked until the first transaction completes. The disadvantage is that the slower the transaction holding the lock runs, the longer the transaction waiting to be unlocked will be blocked. And deadlocks are easy to create (there is a previous article on deadlock)!

This article introduces three ways that redis can implement cas transactions and addresses the following virtualization problems:
Maintain 1 value, set to current time if the value is less than the current time; If the value is greater than the current time, set to the current time +30. The code for a simple single-threaded environment is as follows:


#  Initialize the 
r = redis.Redis()
if not r.exists("key_test"):
  r.set("key_test", 0)

def inc():
  count = int(r.get('key_test')) + 30 #1
  #  Set to the current time if the value is less than the current time 
  count = max(count, int(time.time())) #2
  r.set('key_test', count) #3
  return count

Very simple 1 piece of code that runs happily in a single-threaded environment, but clearly cannot be ported to a multi-threaded or multi-process environment (processes A and B run to #1 at the same time, get the same count value, then run #2#3, resulting in a total increase of only 30 in count value). And in order to run in a multi-process environment, we need to introduce a few other things.

py-redis comes with its own transaction operations

redis has several commands related to transactions, multi,exec,watch. With these commands, you can 'package multiple commands and then execute them once, sequentially, without being terminated'. The transaction starts with MULTI and fires an event after EXEC is executed. In addition, we need WATCH, watch can monitor any number of keys, when calling EXEC to execute a transaction, if any one key is modified, the entire transaction will not execute.

Here is the code that USES redis's own transactions to solve the cas problem.


class CasNormal(object):
  def __init__(self, host, key):
    self.r = redis.Redis(host)
    self.key = key
    if not self.r.exists(self.key):
      self.r.set(self.key, 0)

  def inc(self):
    with self.r.pipeline() as pipe:
      while True:
        try:
          # monitoring 1 a key Is thrown if it has been modified during execution WatchError
          pipe.watch(self.key)
          next_count = 30 + int(pipe.get(self.key))
          pipe.multi()
          if next_count < int(time.time()):
            next_count = int(time.time())
          pipe.set(self.key, next_count)
          pipe.execute()
          return next_count
        except WatchError:
          continue
        finally:
          pipe.reset()

The code is not complicated, it introduces multi,exec,watch mentioned earlier, if you are familiar with transaction operation, you can easily see that this is an optimistic lock operation (let's assume that no one is competing, every time they go to get the data, they will not lock, but someone will change it). Optimistic locking can be weak at high concurrency, as the performance comparison at the end of this article shows.

Use the pessimistic lock based on redis

Pessimistic lock, is very pessimistic lock, every time to take data will assume that others also want to take, to lock up, use up the lock to release. redis itself does not implement a pessimistic lock, but we can implement a pessimistic lock with redis first.

ok, now that we have a pessimistic lock, we have the confidence to do things, according to the above code, we can just add @synchronized annotation to ensure that only 1 process is executing at the same time. Here is a solution based on pessimistic locking.


lock_conn = redis.Redis("localhost")

class CasLock(object):
  def __init__(self, host, key):
    self.r = redis.Redis(host)
    self.key = key
    if not self.r.exists(self.key):
      self.r.set(self.key, 0)

  @synchronized(lock_conn, "lock", 10)
  def inc(self):
    next_count = 30 + int(self.r.get(self.key))
    if next_count < int(time.time()):
      next_count = int(time.time())
    self.r.set(self.key, next_count)
    return next_count

The code looks a lot less (thanks to the introduction of synchronized...)

Implementation based on lua script

The above two kinds of methods are to use the lock to achieve, the realization of the lock will always appear the problem of competition, the difference is nothing but the emergence of competition how to do the problem. Using the implementation of the redis lua script, you can directly treat this cas operation as one < b > Atomic operation < /b > .

As we know, the series 1 operations of redis itself are atomic operations, and redis executes all received commands in sequence. Look at the code


class CasLua(object):
  def __init__(self, host, key):
    self.r = redis.Redis(host)
    self.key = key
    if not self.r.exists(self.key):
      self.r.set(self.key, 0)
    self._lua = self.r.register_script("""
    local next_count = redis.call('get',KEYS[1]) + ARGV[1]
    ARGV[2] = tonumber(ARGV[2])
    if next_count < ARGV[2] then
      next_count = ARGV[2]
    end
    redis.call('set',KEYS[1],next_count)
    return tostring(next_count)
        """)

  def inc(self):
    return int(self._lua([self.key], [30, int(time.time())]))

The script is registered here, and you can use it later. There are many articles about redis lua script, those who are interested in it can go to search and see, and I will not repeat it here.

The performance comparison

The test here is just a very simple test (but still can see the effect), the test switch is your own development machine, the number to see the size of the line.

The performance of 3 operations in a single thread, 5 threads, 10 threads and 510 threads for 1000 operations was measured respectively. The time was as follows


     optimistic Lock pessimistic lock  lua
1thread       0.43       0.71 0.35
5thread       5.80       3.10 0.62
10thread      17.80       5.60 1.30
50thread      245.00       29.60 6.50

In turn, the optimistic lock of redis's own transaction implementation, the pessimistic lock based on redis implementation, and the lua implementation.

Before getting into pessimistic locks and optimistic locks, it's worth noting that the tests here are not very fair to optimistic locks, which assume that there won't be a lot of concurrency. In the case of a single thread, pessimistic locking is less than 1. In a single thread, there is no competition, and pessimistic locking time is long only because there is one more redis network interaction. As the number of threads increases, the performance of pessimistic locks improves. Pessimistic locks, after all, were created to solve this highly concurrent and competitive environment. At 50 threads, optimistic locking takes 0.245 seconds to implement a single operation, which is so horrible that it would be almost unusable in a production environment.

As for the Performance of the lua, it's incredibly fast, almost linearly increasing. (In the case of 50 threads, the average 1000 completion time is 6.5es111EN, in other words, 50 * 1000 cas operations were performed in 6.5 seconds).


Related articles: