Solve some pits in Python process pool Pool

1 from multiprocessing import Pool, Queue.

Among them, Queue does not play a role in Pool, and the specific reasons are unknown.

Solution:

If you want to create a process with Pool, you need to use Queue in multiprocessing. Manager (),

Unlike Queue in multiprocessing

q=Manager().Queue()#Manager In Queue To cooperate Pool
po = Pool() #  Infinitely many processes 

2 Use process pool to call io read and write operations in the process.

For example:

p=Pool()
q=Manager().Queue()
with open('/home/cctv/data/stage_file/stage_{}.txt'.format(int(time.time())), 'w') as w1:
 p.apply_async(write_json, args=(video_path, 0,0.6,w1,q,i[0],))

In this way, the process will not be completed, and w can only be put into specific functions, and cannot be called through parameters

Supplement: python3 Process Pool pool Use and Precautions

1. The main purpose of using process pool in python is to process tasks in parallel and shorten running time

2. Frequently used methods: apply () and map () are synchronized; Asynchronous ones are apply_async (), map_async ()

3. Look at a few small examples first

import time 
from multiprocessing import Pool 
test = [1,2,3,4,5,6,7,8]
def run(fn):
 time.sleep(1)
 return fn*fn
s = time.time()
for i in test:
 run(i)
e = time.time()
print(' Direct circulation   Execution time :',e - s)
pool = Pool(8)
s = time.time()
for i in test: 
 pool.apply(run, (i,))
e = time.time()
print('apply  Execution time :',e - s)
pool1 = Pool(8)
s = time.time()
res = []
for i in test: 
 r = [pool1.apply_async(run, (i,))]
 res.append(r)
pool1.close()
pool1.join()
e = time.time()
print([i.get() for i in r])
print('apply_async  Execution time :',e - s)
 
pool2 = Pool(8)
r = pool2.map(run,test)
pool2.close()
pool2.join() 
e1 = time.time()
print(r)
print('map Execution time :',e1 - e)
pool3 = Pool(8)
pool3.map_async(run,test)
pool3.close()
pool3.join() 
e1 = time.time()
print('map_async Execution time :',e1 - e)

Execution results

 Direct circulation   Execution time : 8.004754781723022
apply  Execution time : 8.016774654388428
[64]
apply_async  Execution time : 1.1128439903259277
[1, 4, 9, 16, 25, 36, 49, 64]
map Execution time : 1.181443452835083
map_async Execution time : 2.3679864406585693

In addition, in writing code, it also involves 1 problem of variables. You need to lock it ~