asyncio asynchronous programming learning in python

2021-10-25 07:06:22
OfStack

1. If you want to learn asyncio, you must first understand the coordination process

Significance of Ctrip:

Computational operations, which are performed by switching back and forth with synergy, make no sense. Switching back and forth and saving state will reduce performance. IO type operation, using co-process in IO waiting time to switch to perform other tasks, when IO operation after the end of automatic callback, then will greatly save resources and provide performance, thus achieving asynchronous programming (do not wait for the end of the task to perform other code

2. Similarities and differences between cothreading and multithreading:

Common ground:

All of them are concurrent operations, and only one thread can be executed at the same time point of multi-threads, and only one task can be executed at the same time point of co-threads;

Differences:

Multithreading is to achieve concurrent effect by switching threads when I/O is blocked. Under what circumstances to switch threads is determined by the operating system. Developers don't have to worry, but it will cause competition conditions (race condition);

Co-process, only one thread, when I/O is blocked, the concurrent effect is achieved by switching tasks within the thread. Under what circumstances to switch tasks is decided by the developer, and there will be no competition conditions (race condition); Thread switching of multi-threads is more overhead than task switching of co-threads.
For developers, multi-threaded concurrent code is easier to write than co-threaded concurrent code.

1 Under normal circumstances, the processing efficiency of co-threaded concurrency is higher than that of multithreaded concurrency.

3. greenlet implementation coordination process

greenlet is used to create synergies, switch is used for switching between coroutines. A certain coroutine can be interrupted by other coroutines at any time through switch function. Instead of executing other routines, the interrupt scene of the current routines will be retained, and the 1-denier interrupt routines will get the execution right of cpu again. First, they will resume the scene and then continue to execute from the interrupt. The routines under this mechanism are synchronous and cannot be concurrent

pip install greenlet


import time
import greenlet
 
 
def func1():
  print("func11")
  gr2.switch()
  time.sleep(1)
  print("func22")
  gr2.switch()
 
 
def func2():
  print("func33")
  gr1.switch()
  time.sleep(1)
  print("func44")
 
 
start = time.time()
gr1 = greenlet.greenlet(func1)
gr2 = greenlet.greenlet(func2)
gr1.switch()
end = time.time()
print(end - start)

4. yield keyword implementation


def func1():
  yield 1
  yield from func2()
  yield 3
 
 
def func2():
  yield 2
  yield 4
 
 
ff = func1()
for item in ff:
  print(item)

5. gevent co-process

(1) gevent implementation co-process

pip install gevent


from greenlet import greenlet
from time import sleep
def func1():
  print(" Synergetic process 1")
  sleep(2)
  g2.switch()
  print(" Synergetic process 1 Resume operation ")
 
def func2():
  print(" Synergetic process 2")
  sleep(1)
  g3.switch()
def func3():
  print(" Synergetic process 3")
  sleep(1)
  g1.switch()
 
if __name__ == '__main__':
  #  Use greenlet To create 3 Individual cooperation process 
  g1 = greenlet(func1)
  g2 = greenlet(func2)
  g3 = greenlet(func3)
  # print(g1)
  g1.switch() #  Let Xiecheng g1 Fetch preemption cpu Resources

(2) gevent realizes asynchronous co-process


#  After a co-process is created, multiple co-processes are executed synchronously by default 
#  We can join in monkey Patch, which converts synchronous co-process into asynchronous co-process 
from gevent import monkey #  Note: monkey The introduction of must precede other modules 
 
monkey.patch_all() #  Use monkey To the entire co-process queue, add 1 Non-blocking I/O Make them asynchronous co-processes 
import time
import requests
import gevent
 
headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
 
 
def func(url, i):
  print(" Synergetic process %d Open it! " % i)
  res = requests.get(url=url, headers=headers)
  html = res.text
  print(" Synergetic process %d At the end of execution, the obtained response body size is: %d" % (i, len(html)))
 
 
if __name__ == '__main__':
  start = time.time()
  urls = [
    "https://www.baidu.com/",
    "https://www.qq.com/",
    "https://www.sina.com.cn",
    "https://www.ifeng.com/",
    "https://www.163.com/"
  ]
  #  Create 5 Individual cooperation programs are respectively on the above 5 Visit 10 websites 
  g_list = []
  for i in range(len(urls)):
    g = gevent.spawn(func, urls[i], i)
    g_list.append(g)
    # func(urls[i], i)
  gevent.joinall(g_list)
  end = time.time()
  print(end - start)

6. asyncio module realizes asynchronous co-process

Used in python 3.4 and later, asyncio is great because it automatically switches to other tasks when IO is encountered


import time
import asyncio
 
 
@asyncio.coroutine
def func1():
  print(1)
  yield from asyncio.sleep(1) #  Encounter IO Time-consuming operation, automatically switching to tasks Other tasks in the 
  print(2)
 
 
@asyncio.coroutine
def func2():
  print(3)
  yield from asyncio.sleep(1) #  Encounter IO Time-consuming operation, automatically switching to tasks Other tasks in the 
  print(4)
 
 
tasks = [
  asyncio.ensure_future(func1()),
  asyncio.ensure_future(func2())
]
 
start = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print(end - start)

7. asyc & The await keyword implements asynchronous programming (now recommended)

Available in python 3.5 and later


import time
import asyncio
 
 
async def func1():
  print(1)
  await asyncio.sleep(1)
  print(2)
 
 
async def func2():
  print(3)
  await asyncio.sleep(1)
  print(4)
 
 
tasks = [
  asyncio.ensure_future(func1()),
  asyncio.ensure_future(func2())
]
 
start = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
end = time.time()
print(end - start)

7.1 Event Loop

Event loop, you can think of it as an while loop. This while loop runs periodically and executes 1 任务 That terminates the loop under certain conditions.

Pseudocode:


#  Pseudocode 
 Task list  = [  Mission 1,  Mission 2,  Mission 3,... ]
while True:
   List of Executable Tasks, List of Completed Tasks  =  Go to the task list to check all tasks, and set ' Executable ' And ' Completed ' Returns the task of 
  for  Ready task  in  List of Ready Tasks :
     Perform a ready task 
  for  Completed tasks  in  List of completed tasks :
     Remove from the task list   Completed tasks 
   If   Task list   Terminate the loop if all the tasks in the

7.2 Synergetic and asynchronous programming

Synergetic function, defined as async def The function of.

Synergetic object, the object returned by calling the Synergetic function.


#  Definition 1 Individual coequation function 
async def func():
  pass
#  Call the synchronic function and return 1 Individual cooperative object 
result = func()

Note: When you call a synergetic function, the internal code of the function does not execute, only one synergetic object is returned.

7.3 Basic Applications

In the program, if you want to execute the internal code of the synergetic function, you need 事件循环 And 协程对象 Cooperation can be achieved, such as:


import asyncio
async def func():
  print(" Synergetic internal code ")
#  Call the synchronic function and return 1 Individual co-process object. 
result = func()
#  Mode 1
# loop = asyncio.get_event_loop() #  Create 1 Event loops 
# loop.run_until_complete(result) #  Submits the synchro as a task to the task list of the event loop, and terminates after the synchro executes. 
#  Mode 2
#  Essential way 1 Yes 1 Like, inside first   Create an event loop   Then execute  run_until_complete , 1 A simple way to write. 
# asyncio.run  Function in  Python 3.7  Add to  asyncio  Module, 
asyncio.run(result)

This process can be simply understood as: 协程 Add as a task to the 事件循环 The task list of, and then the event loop detects the task list in the list 协程 Is ready (understood as ready by default), and if so, executes its internal code.

7.4 await Keyword

await is a key word that can only be used in the co-process function. It is used to suspend the current co-process (task) when encountering IO operation. In the process of suspending the current co-process (task), the event cycle can execute other co-processes (tasks). When the current co-process IO processing is completed, it can switch back to execute the code after await again.

await + Waitable Objects (Synergetic Objects, Future Objects, Task Objects)

Example 1: await + Synergetic Object


import asyncio
 
 
async def func1():
  print("start")
  await asyncio.sleep(1)
  print("end")
  return "func1 Complete execution "
 
 
async def func2():
  print("func2 Start execution ")
  # await Keyword can be followed by waitable objects (synergetic objects, Future Object, Task Object )
  response = await func1()
  print(response)
  print("func2 Complete execution ")
 
 
asyncio.run(func2())

Example 2: You can use the await keyword more than once in a synergetic function


def func1():
  yield 1
  yield from func2()
  yield 3
 
 
def func2():
  yield 2
  yield 4
 
 
ff = func1()
for item in ff:
  print(item)

7.5 task Object

Tasks is used for concurrent scheduling, through asyncio.create_task(协程对象) To create an Task object, which allows the co-process to join the event loop and wait for it to be scheduled for execution. In addition to using asyncio.create_task() Function, you can also use a low-level loop.create_task() Or async def0 Function. Manual instantiation of Task objects is not recommended.

Essentially, it encapsulates the synergy object as an task object, and immediately joins the synergy into the event loop, while tracking the status of the synergy.

Note: asyncio.create_task() Function is added in Python 3.7. Prior to Python 3.7, you can switch to a lower-level asyncio.ensure_future() Function.

Example 1:


def func1():
  yield 1
  yield from func2()
  yield 3
 
 
def func2():
  yield 2
  yield 4
 
 
ff = func1()
for item in ff:
  print(item)

Example 2: It is still used more


def func1():
  yield 1
  yield from func2()
  yield 3
 
 
def func2():
  yield 2
  yield 4
 
 
ff = func1()
for item in ff:
  print(item)

Example 3:


import asyncio
 
 
async def func():
  print(" Execute the internal code of the synergetic function ")
  #  Encounter IO Operation suspends the current program (task), and so on IO Continue after the operation is completed. When the current program is suspended, the event loop can execute other programs (tasks). 
  response = await asyncio.sleep(2)
  print("IO The request ends with the result: ", response)
 
 
coroutine_list = [func(), func()]
#  Error: coroutine_list = [ asyncio.create_task(func()), asyncio.create_task(func()) ]
#  You can't directly here  asyncio.create_task Because the Task Join the task list of the event loop immediately, 
#  But at this time, the event loop has not been created, so an error will be reported. 
#  Use asyncio.wait Encapsulate the list as 1 Individual program, and call asyncio.run Implement two co-processes 
# asyncio.wait Internally, it executes for each program in the list ensure_future Encapsulated as Task Object. 
done, pending = asyncio.run(asyncio.wait(coroutine_list))

Summary:

In the program, as long as you see async And await Keyword, its interior is asynchronous programming based on co-programming, which is to perform other tasks through one thread waiting time in IO, thus realizing concurrency.

If I/O is intensive and I/O requests are time consuming, use synergy.
If I/O is intensive and I/O requests are fast, use multithreading.
If it is computationally intensive, consider using multi-core CPU and using multiple processes.

Above is python in asyncio asynchronous programming learning details, more about python using asyncio information please pay attention to other related articles on this site!