Summary of python Practice of yeild async azwait and Synergy
- 2021-12-05 06:42:08
- OfStack
1. Brief analysis of the use of yield
yield is a generator generator that returns an interable object.
This object has an next () method, and you can see what the next element is through next ().
1. interable objects, objects that can be traversed, such as list, str, tuple, dict, file, xrange, etc.
2. What is the function of 2. yield? It is only a method for obtaining intermediate variables in the loop, saving the desired variables every time using yield until the end of the loop, and getting an generator object at the end of the loop
3. Why use yield? Using yield, a function is rewritten into generator, which has the ability of iteration. Compared with using an instance of a class to save the state to calculate the next value that needs iteration, the code is more concise and the execution process is 10 points simple.
4. How to judge the type of yield?
def fab(max):
n, a, b = 0, 0, 1
while n < max:
yield b # Use yield
# print b
a, b = b, a + b
n = n + 1
for n in fab(5):
print n
fab is not generator, fab (5) is generator.
For example, the definition of a class is different from the instance of a class.
>>>import types
>>> isinstance(fab, types.GeneratorType)
False
>>> isinstance(fab(5), types.GeneratorType)
True
fab is not iterative, while fab (5) is iterative.
>>>from collections import Iterable
>>> isinstance(fab, Iterable)
False
>>> isinstance(fab(5), Iterable)
True
5. The application of 5. yield in file reading?
If the byte uses read () to read 1 file, it will result in unpredictable memory footprint. A good idea is to use yield, a fixed-length buffer to read files continuously, generating an iterative generator for reading files.
def read_file(fpath):
BLOCK_SIZE = 1024
with open(fpath, 'rb') as f:
while True:
block = f.read(BLOCK_SIZE)
if block:
yield block
else:
return
2. Use of async and await
1. What are processes, co-processes and asynchronous?
What is Synergy?
A user-level lightweight thread with its own register context and stack.
Save the register and stack elsewhere when switching, and restore the previously saved register context and stack when returning.
Why use Synergy?
Mainstream languages adopt multithreaded concurrency. Thread-related concepts are preemptive multitasking and cooperative multitasking.
Whether it is multi-process or multi-threaded, every time it blocks, the switch falls into a system call.
CPU runs the scheduler of the operating system, and the scheduler decides which 1 process (thread) to run.
Threads handle synchronization problems very carefully, while co-threads do not have this problem at all.
For CPU, multi-thread is a single thread, and CPU does not consider scheduling and switching context, thus saving the switching overhead of CPU. The reason why co-threading is better than multithreading.
How to use Synergy?Under multi-process + co-process, it avoids the overhead of CPU switching, and can make full use of multiple CPU. This way has great efficiency improvement for crawlers with large data volume and file reading and writing.
2. How to deal with the number of url in 200W and save all url?
Single process + single thread Single process + multithreading: Open 10 threads, and the speed cannot be increased by 10 times. Thread switching is overhead, and it is impossible to create threads indefinitely. Multi-process + multi-thread: Each process of multi-process occupies 1 CPU, and multi-thread bypasses the blocking time to a certain extent, so it is more efficient than single-process multi-thread. Synergetic process
3. await and gather using async
await accepts one list and returns two lists, done and pending. done is a completed process, and pending is a process that is still running. Get the completed result via. result () gather accepts the program as gather (cro1, cro2, cro3, cro4...), and returns a program that combines so many tasks.Usage of async: https://blog.csdn.net/qq _ 29785317/article/details/103294235
async def func1(num):
print('--func1 start--')
await asyncio.sleep(num)
print('--func1 done--')
return 'func1 ok'
async def func2(num):
print('--func2 start--')
await asyncio.sleep(num)
print('--func2 done--')
return 'func2 ok'
async def main():
task1 = asyncio.ensure_future(func1(3))
task2 = asyncio.ensure_future(func2(5))
tasks = [task1, task2]
res = await asyncio.gather(*tasks)
return res
# done, pending = await asyncio.wait(tasks)
# for t in done:
# print(t.result())
# print(done)
# print(pending)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main())
print(result)
```python
--func1 start--
--func2 start--
--func1 done--
--func2 done--
['func1 ok', 'func2 ok']
3. Understanding of Synergy
1. The process of coordination
yield is the way to control the flow.
Like Receiver 1, yield is a generator that needs to be activated before it can be used.
>>> def simple_corotine():
... print('---->coroutine started')
... x = yield # There is a received value, so it is the same as the generator 1 Sample, need to activate first , Use next
... print('---->coroutine recvied:',x)
...
>>> my_coro = simple_corotine()
>>> my_coro
<generator object simple_corotine at 0x0000000000A8A518>
>>> next(my_coro) # First activate the generator and execute it to yield val Statement # Or use send(None) You can also activate the generator
---->coroutine started
>>> my_coro.send(24) # To which a value is passed, x = yield
---->coroutine recvied: 24
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration # An error is reported when the generator completes execution
2. Four states of co-process in operation
GEN_CREATE: Waiting for execution to begin
GEN_RUNNING: The interpreter is executing, and this state 1 is generally invisible
GEN_SUSPENDED: Pause at yield expression
GEN_CLOSED: End of execution
>>> def averager():
... total = 0.0
... count = 0
... aver = None
... while True:
... term = yield aver
... total += term
... count += 1
... aver = total/count
...
>>> coro_avg = averager()
>>> coro_avg.send(None)
>>> coro_avg.send(10)
10.0
>>> coro_avg.send(20)
15.0
>>> coro_avg.send(30)
20.0
>>> coro_avg.send(40)
25.0
At the end of each loop, it is paused at yield until the next parameter is passed in.
3. Pre-activate the decorator of the coordination process (customize the activation mode)
@ What does the decorator do? Decorate the original function and add a new function and mode to the crude oil function.
Why can @ realize the function of decorator? Functions are also objects, and functions can be passed to pinch functions as arguments.
>>> def coro_active(func):
... def inner(*args,**kwargs):
... gen = func(*args,**kwargs)
... next(gen) #gen.send(None)
... return gen
... return inner
...
>>> @coro_active
... def averager():
... total = 0.0
... count = 0
... aver = None
... while True:
... term = yield aver
... total += term
... count += 1
... aver = total/count
...
>>> coro_avg = averager()
>>> coro_avg.send(10) 10.0 >>> coro_avg.send(20) 15.0 >>> coro_avg.send(30) 20.0
4. Terminate coagulation and exception handling
When an error occurs in the next function or send function of the coprogram, the coprogram will be terminated.
You need to create an exception catch to handle the exception situation of the co-process and close the current co-process.
5. Let the program return a value
Usage of yield
Summarize
This article is here, I hope to give you help, but also hope that you can pay more attention to this site more content!