MELIAE is used in Python to analyze program memory usage instances

Write the DHT protocol search procedures, these days to optimize the speed of discovery is really much faster. However, a new problem appeared, the memory directly soared, I opened ten crawlers to occupy 800m of memory. At first, I thought there were too many nodes, so I found a few minor problems to modify, but found it useless. Later, I went to the Internet to look up python memory analysis tools, looked up a bit of information found that python has a meliae library operation is very convenient, on the use of analysis, found that there is not too many nodes for the reason 0 0, is to save the sent t_id, used to mark the message back is that a dictionary is too large.

It is very easy to locate the number and size of an object from the results of the analysis. At first, I thought it was because a lot of query information was sent, and the elements in the dictionary were not released due to no return, so I used the expiration time to judge and delete them. The findings are small, but not very significant, as if less than a few tens of 100 meters. Later, I reduced the time to find a random hash, which used to be 1 minute, but I changed it to the first time! I don't know what the reason is. I'm just going to look up the hash, ask the nodes, then go back and ask the nodes in it, and it's going to get more and more, but I don't understand how it's going to get so many that there's 600,000 in a minute. That's how many objects were left unfreed. After this footprint is reached, it is basically unchanged, with a small, slow increase, because there are other programs open, and it is not certain that the addition of other objects to these programs is the cause. Test the dump in stages.

Just install PIP install meliae. I haven't updated the project in a long time. I don't know if there are any good alternatives, but it works well.

Dump memory into a file

 from meliae import scanner

 scanner.dump_all_objects('/tmp/dump%s.txt' % time.time())

Analysis document:

 from meliae import loader

 # loading dump file 

 om = loader.load('/opt/log/dump.txt')

 # Calculate the Objects Reference relation of 

 om.compute_parents()

 # Get rid of each object Instance the _dict_ attribute 

 om.collapse_instance_dicts()

 # Analyze the memory footprint 

 om.summarize()

The field meaning is as follows:
Index: the number of a row
Count: the total number of objects of this type
%(Count) : the percentage of the total number of objects of this type in the total number of objects of all types
Size: total number of bytes of objects of this type
%(Size) : the percentage of the total number of bytes of objects of this type in the total number of bytes of objects of all types
Cum: %(Size) after cumulative row index
Max: the number of bytes in the largest object of this type
Kind: type

Analyze an object to find its references

 # Get all of POP3ClientProtocol object 

 p = om.get_all('POP3ClientProtocol')

 # Look at the first object 

 p[0]

 # You can view all references to this object 

 p[0].c

 # See who references the object 

 p[0].p