Python multiprocessing.manager introduction and instances (sharing data between processes)

  • 2020-04-02 14:24:45
  • OfStack

Python provides a higher level of encapsulation beyond the basic queue, pipe, and value+array for sharing data between processes. These high-level interfaces can be easily used with the multiprocessing.manager.

Manager () returns the object Manager controls a server process, this process include python objects can be other processes to access through proxies. In order to achieve multi-process data communication and security.

Support Manager of type list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, the Value and the Array.

1) dict of the Manager, using list


import multiprocessing
import time def worker(d, key, value):
    d[key] = value if __name__ == '__main__':
    mgr = multiprocessing.Manager()
    d = mgr.dict()
    jobs = [ multiprocessing.Process(target=worker, args=(d, i, i*2))
             for i in range(10)
             ]
    for j in jobs:
        j.start()
    for j in jobs:
        j.join()
    print ('Results:' )
    for key, value in enumerate(dict(d)):
        print("%s=%s" % (key, value))
       
# the output is :
# Results:
# 0=0
# 1=1
# 2=2
# 3=3
# 4=4
# 5=5
# 6=6
# 7=7
# 8=8
# 9=9

The above is a usage example of manager.dict.

2) the namespace object has no public methods, but it has writable properties.

However, when the proxy of the namespace returned by the manager is used, the _ attribute value belongs to the proxy and has nothing to do with the original namespace.


>>> manager = multiprocessing.Manager()
>>> Global = manager.Namespace()
>>> Global.x = 10
>>> Global.y = 'hello'
>>> Global._z = 12.3    # this is an attribute of the proxy
>>> print(Global)
Namespace(x=10, y='hello')


Related articles: