Of Stack

A brief understanding of the use and function of python gevent co process

  • 2021-07-24 11:24:23
  • OfStack
Directory Introduction
yield
gevent
Note

Brief introduction

There is no switching overhead. Because subroutine switching is not thread switching, but controlled by the program itself, there is no overhead of thread switching, so the execution efficiency is high,

No locking mechanism is required. Because there is only one thread, and there is no conflict of writing variables at the same time, the shared resources are controlled without locking in the co-process, and only the state is judged, so the execution efficiency is much higher than that of multi-thread

The support of Python for co-process is still very limited, and yield used in generator can realize co-process to a certain extent.

yield

In the traditional producer-consumer model, one thread writes messages and one thread fetches messages, and queues and waits are controlled by locking mechanism, but one may be deadlocked if one is not careful.

If you use co-process instead, after the producer's production message, you can jump to the consumer to start execution directly through yield. After the consumer completes the execution, you can switch back to the producer to continue production, which is extremely efficient

Code 


import time

def consumer():
  r = ''
  while True:
    n = yield r
    if not n:
      return
    print('[CONSUMER] Consuming %s....' % n)
    r = '200 OK'

def produce(c):
  c.next()
  n = 0
  while n < 5:
    n = n + 1
    print('[PRODUCER] Producing %s...' % n)
    r = c.send(n)
    print('[PRODUCER] Consumer return: %s\n' % r)
  c.close()

if __name__=='__main__':
  c = consumer()
  produce(c)

Results 


[PRODUCER] Producing 1...
[CONSUMER] Consuming 1....
[PRODUCER] Consumer return: 200 OK

[PRODUCER] Producing 2...
[CONSUMER] Consuming 2....
[PRODUCER] Consumer return: 200 OK

[PRODUCER] Producing 3...
[CONSUMER] Consuming 3....
[PRODUCER] Consumer return: 200 OK

[PRODUCER] Producing 4...
[CONSUMER] Consuming 4....
[PRODUCER] Consumer return: 200 OK

[PRODUCER] Producing 5...
[CONSUMER] Consuming 5....
[PRODUCER] Consumer return: 200 OK

Analysis

First call c. next () to start the generator Then, 1 denier produces something and switches to consumer for execution through c. send (n) consumer gets the message through yield, processes it, and sends the result back through yield produce gets the result processed by consumer and continues to produce the next message

The whole process is unlocked and executed by one thread, and producer and consumer write to complete the task, so it is called co-process

gevent

Python provides basic, but incomplete, support for synergies through yield. The third party's gevent provides more perfect co-process support for Python

gevent is the third party library, which realizes synergy through greenlet. Its basic idea is:

When an greenlet encounters an IO operation (such as accessing the network), it automatically switches to other greenlet, waits until the IO operation is completed, and then switches back to continue execution at an appropriate time. Because the operation of IO is very time-consuming, the program is often in a waiting state. With gevent, we can automatically switch the co-program, which ensures that there is always greenlet running instead of waiting for IO. 


import gevent

def f(n):
  for i in range(n):
    print gevent.getcurrent(), i

g1 = gevent.spawn(f, 5)
g2 = gevent.spawn(f, 5)
g3 = gevent.spawn(f, 5)

g1.join()
g2.join()
g3.join()

Results 


<Greenlet at 0x7f7216efbe10: f(5)> 0
<Greenlet at 0x7f7216efbe10: f(5)> 1
<Greenlet at 0x7f7216efbe10: f(5)> 2
<Greenlet at 0x7f7216efbe10: f(5)> 3
<Greenlet at 0x7f7216efbe10: f(5)> 4
<Greenlet at 0x7f720f54e0f0: f(5)> 0
<Greenlet at 0x7f720f54e0f0: f(5)> 1
<Greenlet at 0x7f720f54e0f0: f(5)> 2
<Greenlet at 0x7f720f54e0f0: f(5)> 3
<Greenlet at 0x7f720f54e0f0: f(5)> 4
<Greenlet at 0x7f720f54e190: f(5)> 0
<Greenlet at 0x7f720f54e190: f(5)> 1
<Greenlet at 0x7f720f54e190: f(5)> 2
<Greenlet at 0x7f720f54e190: f(5)> 3
<Greenlet at 0x7f720f54e190: f(5)> 4

It can be seen that the three greenlet are running in turn, not alternately

To make greenlet run alternately, you can hand over control through gevent. sleep () 


import gevent

def f(n):
  for i in range(n):
    print gevent.getcurrent(), i
    gevent.sleep(1)

g1 = gevent.spawn(f, 3)
g2 = gevent.spawn(f, 3)
g3 = gevent.spawn(f, 3)

g1.join()
g2.join()
g3.join()

Results 


<Greenlet at 0x7f74e2179e10: f(3)> 0
<Greenlet at 0x7f74da7cb0f0: f(3)> 0
<Greenlet at 0x7f74da7cb190: f(3)> 0
<Greenlet at 0x7f74e2179e10: f(3)> 1
<Greenlet at 0x7f74da7cb0f0: f(3)> 1
<Greenlet at 0x7f74da7cb190: f(3)> 1
<Greenlet at 0x7f74e2179e10: f(3)> 2
<Greenlet at 0x7f74da7cb0f0: f(3)> 2
<Greenlet at 0x7f74da7cb190: f(3)> 2

It can be seen that the three greenlet are executed alternately

If you change the loop to 1000, so that the number of executions is longer, looking at the process, you can see that there is only one thread.

Of course, in the actual code, it is impossible to use gevent. sleep () to switch the coprogram, but when performing IO operation, gevent switches automatically. The reference code is as follows 


import gevent
from gevent import monkey; monkey.patch_all()
import urllib2

def f(url):
  print 'GET: %s' % url
  resp = urllib2.urlopen(url)
  data = resp.read()
  print '[%d] bytes received from %s\n' %(len(data), url)

gevent.joinall([
gevent.spawn(f, 'http://www.cnblogs.com/kaituorensheng/'),
gevent.spawn(f, 'https://www.python.org/'),
gevent.spawn(f, 'https://www.baidu.com'),
])

Execution results 


GET: http://www.cnblogs.com/kaituorensheng/
GET: https://www.python.org/
GET: https://www.baidu.com
[227] bytes received from https://www.baidu.com

[14667] bytes received from http://www.cnblogs.com/kaituorensheng/

[47348] bytes received from https://www.python.org/

It can be seen that the three url ending sequences are not executed in turn.

Note

With gevent, high concurrency performance can be achieved, but gevent can only run under Unix/Linux, and normal installation and operation are not guaranteed under Windows.

Because gevent is based on the coordination process of IO switching, the most amazing thing is that the Web App code we wrote does not need to introduce gevent package, nor does it need to change any code. Only when we deploy, we use an WSGI server supporting gevent, and immediately get several times of performance improvement.

Related articles: