Examples of multiprocess and multithreaded functionality implemented by Python

  • 2020-10-23 21:04:35
  • OfStack

This article gives an example of Python's multi-process and multi-threading capabilities. To share for your reference, the details are as follows:

Listen to a friend, they are currently developing a testing framework, using python to achieve the distributed system. Although python's execution efficiency is not as high as c and c++, it relies on the power of clusters, which creates a lot of pressure.

And the way to get a rough idea of it is

1. A master control machine is responsible for scheduling, such as parameters of execution

2. There are more than one n executor, and each executor deploys one xmlRPC server of python. The master controller calls rpccall, and then the executor executes. In rpccall, there will be some processes in fork1, and each process will create another thread to call the test method. So, it's very extensible.

As for rpc call of python, I haven't been in contact with it before, so I don't know much about it. I found that google has been used for 1 time, which is very simple. I took an online example, as follows: first deploy 1 rpcServer


from SimpleXMLRPCServer import SimpleXMLRPCServer
def add(a , b): 
  return a+bserver = SimpleXMLRPCServer(("10.249.192.38", 8000))# Don't use it here localhost , otherwise only native access 
server.register_function(add)
server.serve_forever()

Client:


from xmlrpclib import Server
Proxyserver = ServerProxy("http://localhost:8000")
try: ret = server.add(30,90) print 'result:', ret print 'result type:', type(ret)
except
 Exception, e: print "exception",e

It's actually pretty simple.

Then I looked at python's multi-process and multi-threaded methods and wrote an example as follows:


#encoding=utf-8
import sys
import os
import time
import pdb
import httplib
import thread
import threading
constant_p = 0 # Create global variable, number of processes 
constant_s = 0 # Create global variable, number of threads 
mutex_g = threading.RLock() # Create global locks 
def run(count):# This function creates 3 Three threads, called at the same time 3 A different function 
  a = 1
  b = 0
  thread.start_new_thread(test0,(a,b))# The parameters passed in here need to be in the form of tuples, and void The Pointers are pretty much the same 
  thread.start_new_thread(test1,(a,b))
  thread.start_new_thread(test2,(a,b))
def test0(a,b):
  global mutex_g
  global constant_s
  threadid = thread.get_ident()
  mutex_g.acquire()# You need to lock up the thread count or the result will be modified 
  constant_s = constant_s+1
  mutex_g.release()
  print "thread 0 called,and the threadid is:%d"%(threadid)
  sys.exit(0)
def test1(a,b):
  global mutex_g
  global constant_s
  threadid = thread.get_ident()
  mutex_g.acquire()
  constant_s = constant_s+1
  mutex_g.release()
  print "thread 1 called,and the threadid is:%d"%(threadid)
  sys.exit(0)
def test2(a,b):
  global mutex_g
  global constant_s
  threadid = thread.get_ident()
  mutex_g.acquire()
  constant_s = constant_s+1
  mutex_g.release()
  print "thread 2 called,and the threadid is:%d"%(threadid)
  sys.exit(0)
def my_fork():
  global constant_p
  pid = os.fork()#fork1 Child process, child process pid=0 At the same time 2 Five processes will execute my_fork() function 
  if (pid == 0):# The child process executes to this if inside 
    constant_p = constant_s + 1
    run(3)
    time.sleep(5)
    print "total thread is %d"%constant_s# The result is 3 , because the child process creation was pressed 3 A thread 
  elif (pid >0):# The parent process executes to this if inside 
    constant_p = constant_s + 1 run(4)
    time.sleep(5)
    print "total thread is %d"%constant_s# And this result is also 3 , because the process is also created 3 A thread 
  else:
    print "fork error"
    sys.exit(0)
  print "total process is %d"%constant_p# The result is 1, because 2 Only 10 processes executed 1 time 
  constant_p = constant_s + 1
  sys.exit(0)
if __name__ == "__main__":
  my_fork()
  #my_fork()
  #my_fork()

More about Python related topics: interested readers to view this site "Python process and thread skills summary", "Python Socket programming skills summary", "Python data structure and algorithm tutorial" and "Python function using techniques", "Python string skills summary", "Python introduction and advanced tutorial" and "Python file and directory skills summary"

I hope this article has helped you with the Python programming.


Related articles: