The use of gevent module in Python

2021-11-14 06:10:25
OfStack

Directory background
What is Synergy?
What is gevent?
An example of synergy
Q & A
Q: gevent failed to capture time
A: Monkey patch
Practice
Asynchronous requests request
Lock of gevent
Tip

Background

Because of the performance problems of Python threads, running code using multiple threads in Python often fails to achieve the desired results. In actual development, we often have the requirement of high concurrency, which requires our code to run faster and execute more effective logic per unit time, thus reducing useless waiting.

What is Synergy?

We can think of threads as lightweight processes, so you can also understand threads as lightweight threads. Co-process means that when one thread executes A function, it can be interrupted at any time to execute B function, and can be switched freely. However, this process is not a function call, which is like multithreading, but actually a thread. Among them, gevent is a classic implementation of Python.

What is gevent?

gevent is a concurrency framework of Python, which is implemented based on greenlet, and becomes efficient by using epoll event listening mechanism and many other optimizations. The basic idea is that an greenlet is a co-process. When greenlet encounters IO operation, such as accessing the network, it will automatically switch to other greenlet, wait for IO to complete, and then switch back to continue execution. gevent can help us automatically realize this process of co-process switching.

An example of synergy

Code:


import gevent,time

def f1():
    for i in range(5):
        print('function:@@@f1 | NUM: @@@',i)

        #  Blocking here, gevent Will help us switch to other programs ↓ 
        gevent.sleep(0)

def f2():
    for i in range(5):
        print('function:@@@f2 | NUM: @@@',i)

        #  Blocking here, gevent Will help us switch to other programs. 
        gevent.sleep(0)

#  Create two synergetic objects to execute two functions respectively 
xc1=gevent.spawn(f1)
xc2=gevent.spawn(f2)

#  Give the co-workers to gevent To execute 
gevent.joinall([xc1,xc2])

Implementation results:

function:@@@f1 | NUM: @@@ 0
function:@@@f2 | NUM: @@@ 0
function:@@@f1 | NUM: @@@ 1
function:@@@f2 | NUM: @@@ 1
function:@@@f1 | NUM: @@@ 2
function:@@@f2 | NUM: @@@ 2
function:@@@f1 | NUM: @@@ 3
function:@@@f2 | NUM: @@@ 3
function:@@@f1 | NUM: @@@ 4
function:@@@f2 | NUM: @@@ 4

As mentioned above, when gevent helps us execute two coprocesses, first xc1 blocks when executing to gevent. sleep (0), then gevent helps us switch to xc2, and then blocks when xc2 executes to gevent. sleep (0), at this time, gevent helps us switch to xc1.

Q & A

Q: gevent failed to capture time

Code:


import gevent,time

def f1():
    for i in range(5):
        print('function:@@@f1 | NUM: @@@',i)

        #  Pay attention here 
        time.sleep(0.1)

def f2():
    for i in range(5):
        print('function:@@@f2 | NUM: @@@',i)

        #  Pay attention here 
        time.sleep(0.1)

#  Create two synergetic objects to execute two functions respectively 
xc1=gevent.spawn(f1)
xc2=gevent.spawn(f2)

#  Give the co-workers to gevent To execute 
gevent.joinall([xc1,xc2])

Implementation results:

function:@@@f1 | NUM: @@@ 0
function:@@@f1 | NUM: @@@ 1
function:@@@f1 | NUM: @@@ 2
function:@@@f1 | NUM: @@@ 3
function:@@@f1 | NUM: @@@ 4
function:@@@f2 | NUM: @@@ 0
function:@@@f2 | NUM: @@@ 1
function:@@@f2 | NUM: @@@ 2
function:@@@f2 | NUM: @@@ 3
function:@@@f2 | NUM: @@@ 4

As mentioned above, you will find that time. sleep (0.1) consumes time that gevent cannot capture, resulting in the code being serial. Although we created synergy, it did not play an asynchronous role.
What shall I do? Please look at the solution below.

A: Monkey patch

For the time-consuming that cannot be captured, gevent provides us with a monkey patch. When we patch our program with a monkey patch, gevent will help us automatically switch the coordination process when our program encounters any time-consuming operation, thus realizing asynchronous and high concurrency.

Code:


import gevent,time
from gevent import monkey;monkey.patch_all()

def f1():
    for i in range(5):
        print('function:@@@f1 | NUM: @@@',i)

        #  Pay attention here 
        time.sleep(0.1)

def f2():
    for i in range(5):
        print('function:@@@f2 | NUM: @@@',i)

        #  Pay attention here 
        time.sleep(0.1)

#  Create two synergetic objects to execute two functions respectively 
xc1=gevent.spawn(f1)
xc2=gevent.spawn(f2)

#  Give the co-workers to gevent To execute 
gevent.joinall([xc1,xc2])

Implementation results:

function:@@@f1 | NUM: @@@ 0
function:@@@f2 | NUM: @@@ 0
function:@@@f1 | NUM: @@@ 1
function:@@@f2 | NUM: @@@ 1
function:@@@f1 | NUM: @@@ 2
function:@@@f2 | NUM: @@@ 2
function:@@@f1 | NUM: @@@ 3
function:@@@f2 | NUM: @@@ 3
function:@@@f1 | NUM: @@@ 4
function:@@@f2 | NUM: @@@ 4

As above, you will find that the switching of the coordination process has been realized and the problem has been solved perfectly.

Practice

Asynchronous requests request

Code:


from gevent import monkey;monkey.patch_all()
import gevent,time,requests
from urllib3 import disable_warnings

disable_warnings()

def req(url):
    res = requests.get(url,verify=False)
    if res:
        print('URL:{} | CODE:{}!'.format(url,res.status_code))
    else:
        print('URL:{} FAILED!')

xc1=gevent.spawn(req,'https://www.baidu.com')
xc2=gevent.spawn(req,'https://www.gitee.com')
xc3=gevent.spawn(req,'https://www.huaweicloud.com')

gevent.joinall([xc1,xc2,xc3])

Implementation results:

URL:https://www.baidu.com | CODE:200!
URL:https://www.huaweicloud.com | CODE:200!
URL:https://www.gitee.com | CODE:200!

Lock of gevent

Code:


from gevent import monkey;monkey.patch_all()
from gevent.lock import Semaphore
import gevent,time

#  Semaphore set to 1
s1=Semaphore(1)

def f1():
    for i in range(5):
        #  Semaphore -1 That is, get the lock 
        s1.acquire()
        print('function:@@@f1 | NUM: @@@',i)

        #  Semaphore +1 That is, to release the lock 
        s1.release()

        #  Monkey patch helps identify blockage 
        time.sleep(0.1)

def f2():
    for i in range(5):
        #  Semaphore -1 That is, get the lock 
        s1.acquire()
        print('function:@@@f2 | NUM: @@@',i)

        #  Semaphore +1 That is, to release the lock 
        s1.release()

        #  Monkey patch helps identify blockage 
        time.sleep(0.3)

#  Create two synergetic objects to execute two functions respectively 
xc1=gevent.spawn(f1)
xc2=gevent.spawn(f2)

#  Give the co-workers to gevent To execute 
gevent.joinall([xc1,xc2])

Implementation results:

function:@@@f1 | NUM: @@@ 0
function:@@@f2 | NUM: @@@ 0
function:@@@f1 | NUM: @@@ 1
function:@@@f1 | NUM: @@@ 2
function:@@@f2 | NUM: @@@ 1
function:@@@f1 | NUM: @@@ 3
function:@@@f1 | NUM: @@@ 4
function:@@@f2 | NUM: @@@ 2
function:@@@f2 | NUM: @@@ 3
function:@@@f2 | NUM: @@@ 4

As you can see above, gevent can automatically handle locking and blocking. According to the blocking rule, f1 and f2 will be executed alternately, but with the blocking time, because the blocking time of f2 is three times that of f1, the number of times f1 is three times that of f2 in the first six prints, that is, gevent can automatically judge and deal with the coexistence of blocking and locking.

Tip

In the asynchronous development of HTTP, the monkey patch should be played before importing gevent, otherwise there will be an exception.