python implementation of multi process code example
- 2021-01-22 05:13:08
- OfStack
To take full advantage of multi-core CPU resources, Python requires the use of multiple processes in most cases, and Python provides the multiprocessing package for multi-processes. multiprocessing supports sub-processes, synchronization and communication between processes, and provides Process, Queue, Pipe, Lock and other components.
Pioneer child process
The Process class is provided in multiprocessing to generate process instances
Process([group [, target [, name [, args [, kwargs]]]]])
Let's start with a small example:
# -*- coding:utf-8 -*-
from multiprocessing import Process, Pool
import os
import time
def run_proc(wTime):
n = 0
while n < 3:
print "subProcess %s run," % os.getpid(), "{0}".format(time.ctime()) # Gets the current process number and the time it was running
time.sleep(wTime) # Waiting (dormant)
n += 1
if __name__ == "__main__":
p = Process(target=run_proc, args=(2,)) # Application subprocess
p.start() # Running processes
print "Parent process run. subProcess is ", p.pid
print "Parent process end,{0}".format(time.ctime())
Running results:
[
Parent process run. subProcess is 30196
Parent process end,Mon Mar 27 11:20:21 2017
subProcess 30196 run, Mon Mar 27 11:20:21 2017
subProcess 30196 run, Mon Mar 27 11:20:23 2017
subProcess 30196 run, Mon Mar 27 11:20:25 2017
According to the results of the run, the child process is still running after the parent process has finished running, which can cause a zombie (zombie) process.
Typically, when a child process terminates, it notifies the parent process, clears the memory it has occupied, and leaves its own exit message in the kernel. When the parent process learns that the child process is terminated, it retrieves the child's exit information from the kernel. However, if the parent process terminates before the child process, this may cause the child's exit information to remain in the kernel and the child to become a zombie (zombie) process. When a large number of zombie processes accumulate, memory space can be crowded.
What can be done to avoid zombie processes?
When the value deamon is TRUE, the parent process terminates and the process terminates (even if it has not yet finished).
So add p.deamon = true to the above program and see what happens.
# -*- coding:utf-8 -*-
from multiprocessing import Process, Pool
import os
import time
def run_proc(wTime):
n = 0
while n < 3:
print "subProcess %s run," % os.getpid(), "{0}".format(time.ctime())
time.sleep(wTime)
n += 1
if __name__ == "__main__":
p = Process(target=run_proc, args=(2,))
p.daemon = True # join daemon
p.start()
print "Parent process run. subProcess is ", p.pid
print "Parent process end,{0}".format(time.ctime())Execution Result:
[
Parent process run. subProcess is 31856
Parent process end,Mon Mar 27 11:40:10 2017
Again, the problem is that the child process does not finish executing, which is not the desired result. Is there a way to make the parent process finish after the child process has finished executing?
The p.join () method is introduced here, which causes the parent process to execute the following code after the child process finishes executing
# -*- coding:utf-8 -*-
from multiprocessing import Process, Pool
import os
import time
def run_proc(wTime):
n = 0
while n < 3:
print "subProcess %s run," % os.getpid(), "{0}".format(time.ctime())
time.sleep(wTime)
n += 1
if __name__ == "__main__":
p = Process(target=run_proc, args=(2,))
p.daemon = True
p.start()
p.join() # join join methods
print "Parent process run. subProcess is ", p.pid
print "Parent process end,{0}".format(time.ctime())Execution Result:
[
subProcess 32076 run, Mon Mar 27 11:46:07 2017
subProcess 32076 run, Mon Mar 27 11:46:09 2017
subProcess 32076 run, Mon Mar 27 11:46:11 2017
Parent process run. subProcess is 32076
Parent process end,Mon Mar 27 11:46:13 2017
This will allow all the processes to run smoothly.
Define a process as a class
By inheriting the Process class, the process class is defined to implement the run method. The instance p calls the run method automatically by calling p.start().
As follows:
# -*- coding:utf-8 -*-
from multiprocessing import Process, Pool
import os
import time
class Myprocess(Process):
def __init__(self, wTime):
Process.__init__(self)
self.wTime = wTime
def run(self):
n = 0
while n < 3:
print "subProcess %s run," % os.getpid(), "{0}".format(time.ctime())
time.sleep(self.wTime)
n += 1
if __name__ == "__main__":
p = Myprocess(2)
p.daemon = True
p.start() # Automatically call run methods
p.join()
print "Parent process run. subProcess is ", p.pid
print "Parent process end,{0}".format(time.ctime())
The result is the same as in the previous example.
Creating multiple processes
Many times the system needs to create more than one process to improve the utilization of CPU. When the number is small, one instance of Process can be generated manually. Loops may be available when the number of processes is high, but this requires the programmer to manually manage the number of concurrent processes on the system, which can be cumbersome at times. This is where the process pool Pool comes into play. You can limit the number of concurrent processes by passing a parameter, the default being the number of cores for CPU.
For example:
# -*- coding:utf-8 -*-
from multiprocessing import Process,Pool
import os,time
def run_proc(name): ## define 1 A function is used for process calls
for i in range(5):
time.sleep(0.2) # dormancy 0.2 seconds
print 'Run child process %s (%s)' % (name, os.getpid())
# perform 1 This function is all needed 1 The second time
if __name__ =='__main__': # Execute main process
print 'Run the main process (%s).' % (os.getpid())
mainStart = time.time() # Record the start time of the main process
p = Pool(8) # Open up process pool
for i in range(16): # Open up 14 A process
p.apply_async(run_proc,args=('Process'+str(i),))# Each process is called run_proc The function,
#args Represents the arguments passed to the function.
print 'Waiting for all subprocesses done ...'
p.close() # Close the process pool
p.join() # Wait for all initiated processes to finish executing before the main process can proceed
print 'All subprocesses done'
mainEnd = time.time() # Record the end time of the main process
print 'All process ran %0.2f seconds.' % (mainEnd-mainStart) # Main process execution time
Execution Result:
beginning
[
Run the main process (30920).
Waiting for all subprocesses done...
Run child process Process0 (32396)
Run child process Process3 (25392)
Run child process Process1 (28732)
Run child process Process2 (32436)
The last part:
[
Run child process Process15 (25880)
All subprocesses done
All process last 2.49 seconds.
Related Instructions:
There is a limit of 8 concurrent processes in the process pool, and 16 processes will be generated when the program runs. The process pool will automatically manage the number of concurrent processes in the system, and the rest will wait in the queue. The reason for limiting the number of concurrent processes is that it is not possible to have as many concurrent processes on the system as possible, and CPU may spend most of its time scheduling processes rather than performing efficient computations.
In the case of multi-process concurrency, the execution of a single processor to a process is serial. However, the asynchrony of the process is reflected by the unpredictable execution of which process obtains the CPU resource at any given moment (such as the beginning of the execution result, the execution order of each process is variable).
If a single program executes the run_proc function 14 times, it will take at least 16 seconds. With the concurrency of the process, here it takes only 2.49 seconds, showing the advantage of concurrency.