Introduction and usage of 2 Python memory detection tools
- 2020-04-02 13:41:20
- OfStack
When I wrote a program last year, I was not sure how much memory I was using, so I wanted to find a writing tool to print the memory usage of a program or function.
Here is a record of the basic usage of the two memory detection tools that were found last time and will be needed to analyze the memory usage of Python programs in the future.
Memory_profiler module (used with psutil)
Note: psutil module, I love it, it implements many of the main functions of Linux commands, such as: ps, top, lsof, netstat, ifconfig, who, df, kill, free, and so on.
The sample code (https://github.com/smilejay/python/blob/master/py2014/mem_profile.py) :
#!/usr/bin/env python
'''
Created on May 31, 2014
@author: Jay <smile665@gmail.com>
@description: use memory_profiler module for profiling programs/functions.
'''
from memory_profiler import profile
from memory_profiler import memory_usage
import time
@profile
def my_func():
a = [1] * (10 ** 6)
b = [2] * (2 * 10 ** 7)
del b
return a
def cur_python_mem():
mem_usage = memory_usage(-1, interval=0.2, timeout=1)
return mem_usage
def f(a, n=100):
time.sleep(1)
b = [a] * n
time.sleep(1)
return b
if __name__ == '__main__':
a = my_func()
print cur_python_mem()
print ""
print memory_usage((f, (1,), {'n': int(1e6)}), interval=0.5)
Run the above code and the output is:
jay@Jay-Air:~/workspace/python.git/py2014 $python mem_profile.py
Filename: mem_profile.py
Line # Mem usage Increment Line Contents
================================================
15 8.0 MiB 0.0 MiB @profile
16 def my_func():
17 15.6 MiB 7.6 MiB a = [1] * (10 ** 6)
18 168.2 MiB 152.6 MiB b = [2] * (2 * 10 ** 7)
19 15.6 MiB -152.6 MiB del b
20 15.6 MiB 0.0 MiB return a
[15.61328125, 15.6171875, 15.6171875, 15.6171875, 15.6171875]
[15.97265625, 16.00390625, 16.00390625, 17.0546875, 23.63671875, 23.63671875, 23.640625]
Guppy (using Heapy)
Guppy is an umbrella package combining Heapy and GSL with support utilities such as the Glue module that keeps things together.
The sample code (https://github.com/smilejay/python/blob/master/py2014/try_guppy.py) :
#!/usr/bin/env python
'''
Created on May 31, 2014
@author: Jay <smile665@gmail.com>
@description: just try to use Guppy-PE (useing Heapy) for memory profiling.
'''
from guppy import hpy
a = [8] * (10 ** 6)
h = hpy()
print h.heap()
print h.heap().more
print h.heap().more.more
Note the. More usage to output more information.
Run the above program and the output is:
jay@Jay-Air:~/workspace/python.git/py2014 $python try_guppy.py
Partition of a set of 26963 objects. Total size = 11557848 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 177 1 8151560 71 8151560 71 list
1 12056 45 996840 9 9148400 79 str
2 5999 22 488232 4 9636632 83 tuple
3 324 1 283104 2 9919736 86 dict (no owner)
4 68 0 216416 2 10136152 88 dict of module
5 199 1 210856 2 10347008 90 dict of type
6 1646 6 210688 2 10557696 91 types.CodeType
7 1610 6 193200 2 10750896 93 function
8 199 1 177008 2 10927904 95 type
9 124 0 135328 1 11063232 96 dict of class
<91 more rows. Type e.g. '_.more' to view.>
Index Count % Size % Cumulative % Kind (class / dict of class)
10 1045 4 83600 1 11148456 96 __builtin__.wrapper_descriptor
11 109 0 69688 1 11218144 97 dict of guppy.etc.Glue.Interface
12 389 1 34232 0 11252376 97 __builtin__.weakref
13 427 2 30744 0 11283120 97 types.BuiltinFunctionType
14 411 2 29592 0 11312712 98 __builtin__.method_descriptor
15 25 0 26200 0 11338912 98 dict of guppy.etc.Glue.Share
16 108 0 25056 0 11363968 98 __builtin__.set
17 818 3 19632 0 11383600 98 int
18 66 0 18480 0 11402080 98 dict of guppy.etc.Glue.Owner
19 16 0 17536 0 11419616 99 dict of abc.ABCMeta
<81 more rows. Type e.g. '_.more' to view.>
(part of the output is omitted later)
In addition, there is a "PySizer" who also does memory profiling, but not much maintenance.