Example analysis of Python multi process programming techniques

This article analyzes Python multi-process programming techniques in the form of examples, which can help to further Python programming skills. Share with you for your reference. Specific analysis is as follows:

In general, we prefer to use multiple processes in Python because of the limitations of Python threads, such as the fact that multiple threads don't take full advantage of multi-core cpus. But in scenarios like asynchronous UI that don't block, we also use multithreading. This article focuses on the Python multiprocess problem.

Python introduced a multi-process mechanism in 2.6 and provides rich components and apis for writing concurrent applications. The components of the multiprocessing package, such as Process, Queue, Pipe, Lock, etc., provide similar functionality to multithreading. Using these components, you can easily write multiprocess concurrent programs.

The Process

The use of Process is somewhat like java.lang.thread, but a Thread is a Thread. The start method is used to start a process. A simple example:

from multiprocessing import Process
import os
import time
def sleeper(name, seconds):
  print "Process ID# %s" % (os.getpid())
  print "Parent Process ID# %s" % (os.getppid())
  print "%s will sleep for %s seconds" % (name, seconds)
  time.sleep(seconds)

if __name__ == "__main__":
  child_proc = Process(target=sleeper, args=('bob', 5))
  child_proc.start()
  print "in parent process after child process start"
  print "parent process abount to join child process"
  child_proc.join()
  print "in parent process after child process join"
  print "the parent's parent process: %s" % (os.getppid())

To instantiate a Process, you must specify the target and args. The target is the entry method for the new process, which you can think of as the main method. Args is a list of parameters for this method. Starting a process is similar to starting a Thread, and you must call the start method. You can also inherit from Process, override the run method, and implement the Process's logic in the run method. Calling the join method blocks the currently calling process until the called process runs out.
Terminate a process manually by calling the terminate method, which in UNIX sends a SIGTERM semaphore, and in Windows, the TerminateProcess method. It is important to note that the exit processing logic will not be executed, the child processes of the process will not be terminated, they will only become orphan processes.

The Queue

Queue is a multi-process secure Queue, which can be used to transfer data between multiple processes. The put method is used to insert data into the queue. The put method also has two optional parameters: blocked and timeout. If blocked is True (the default) and timeout is positive, the method blocks the time specified by the timeout until the queue has space left. If the timeout occurs, a queue.full exception is thrown. If blocked is False, but the Queue is Full, a queue.full exception is immediately thrown.

The get method reads and removes an element from the queue. Again, the get method has two optional parameters: blocked and timeout. If blocked is True (the default) and timeout is positive, no element is fetched during the wait time and a queue.empty exception is thrown. If blocked is False, there are two situations: if the Queue has a value available, the value is immediately returned; otherwise, if the Queue is Empty, the queue.empty exception is immediately thrown. A sample code for Queue:

from multiprocessing import Process, Queue
def offer(queue):
  queue.put("Hello World")
def test(queue, num):
  queue.put("Hello World: " + str(num))
if __name__ == '__main__':
  q = Queue()
  p1 = Process(target=test, args=(q, 1))
  p1.start()
  p = Process(target=offer, args=(q,))
  p.start()
  p2 = Process(target=test, args=(q, 2))
  p2.start()
  p2 = Process(target=test, args=(q, 3))
  p2.start()
  print q.get()
  print q.get()
  print q.get()
  print q.get()
  print q.close()

Output:

Hello World: 1
Hello World
Hello World: 2
None

Pipes

The Pipe method returns (conn1, conn2) representing two ends of a Pipe. The Pipe method has a duplex parameter, and if the duplex parameter is True(the default value), the Pipe is in full duplex mode, meaning that both conn1 and conn2 are sent and received. Duplex is False, conn1 is only responsible for receiving messages, and conn2 is only responsible for sending messages.

The send and recv methods are methods to send and receive messages, respectively. For example, in full duplex mode, you can call conn1.send to send the message, and conn1.recv to receive the message. If there is no message to receive, the recv method will continue to block. If the pipe is closed, the recv method throws an EOFError.

from multiprocessing import Process, Pipe

def send(conn):
  conn.send("Hello World")
  conn.close()
if __name__ == '__main__':
  parent_conn, child_conn = Pipe()
  p = Process(target=send, args=(child_conn,))
  p.start()
  print parent_conn.recv()

synchronous

The multiprocessing package provides Condition, Event, Lock, RLock, Semaphore and other components for synchronization. Here is an example of using Lock:

from multiprocessing import Process, Lock
def l(lock, num):
  lock.acquire() 
  print "Hello Num: %s" % (num)
  lock.release()
if __name__ == '__main__':
  lock = Lock()
for num in range(20):
  Process(target=l, args=(lock, num)).start()

conclusion

The above is a simple introduction and example of the Python multiprocessing library. If you are familiar with Java multithreading development, do you feel familiar with it? It is similar to the Java Concurrency API.

Interested friends can test run this article's example to deepen their understanding. I believe that this article has a certain reference value for everyone to learn Python programming.