Go pprof Memory metrics meaning memos and case studies

Some of the Go services in the group have been experiencing memory-related issues recently, so I took some time today to take a look at the meaning of the Go pprof memory metrics, in preparation for follow-up questions.

The content is mainly from the Go code comments on these fields, plus their own understanding. You are welcome to correct any misunderstanding.

[

// https://github.com/golang/go/blob/master/src/runtime/mstats.go#L150

// Total number of bytes requested from OS
// Is the sum of the following XxxSys indicators. Contains the sum of heap, stack, and other internal data structures at runtime.
// It is a virtual memory space. Not all are mapped to physical memory.
Sys

/ / ` Sys `
HeapSys

// The number of bytes of objects that are still in use, and objects that are not yet free by GC
// It should be flat and jagged at gc
HeapAlloc

// Number of bytes of object being used.
// One detail is that if more than one span can be contained in one span, as long as one object is in use, then the whole span counts.
// 'HeapInuse' - 'HeapAlloc' is the memory reserved in GC that can be used quickly.
HeapInuse

// Memory has been returned to OS. Memory not reclaimed by the heap.
HeapReleased

// Number of bytes of span not in use.
// This part of memory can be returned to OS and also contains' HeapReleased '.
// can be reapplied and even used as stack memory.
// 'HeapIdle' - 'HeapReleased' is reserved by GC.
HeapIdle

/// ---

/ / and ` HeapAlloc ` 1 sample
Alloc

// Cumulative 'Alloc'
/ / the cumulative mean 1 straight incremental growth since the start with the program, will never fall.
TotalAlloc

// No eggs
Lookups = 0

// Cumulative number of heap objects allocated
Mallocs

// Cumulative number of heap objects freed
Frees

// Number of living objects. See ` HeapAlloc `
// HeapObjects = `Mallocs` - `Frees`
HeapObjects

// ---
// The meaning of Inuse in XxxInuse and Sys in XxxSys are the same as the basic meaning of 'HeapInuse' and 'HeapSys'
// There is no XxxIdle, because it is all included in 'HeapIdle'

// StackSys is basically StackInuse plus system thread level stack memory
Stack = StackInuse / StackSys

// Memory used for the MSpan structure
MSpan = MSpanInuse / MSpanSys

// Memory used for the MCache structure
MCache = MCacheInuse / MCacheSys

// The following are the memory statistics of XxxSys used by the underlying internal data structure
BuckHashSys
GCSys
OtherSys

// ---
// The following is related to GC

// Next time the trigger threshold for GC is reached, it will be GC
NextGC

// unix timestamp for the last GC
LastGC

// Start and end unix timestamps of GC for each cycle
// A cycle may have 0 GC or many GC, and if it is many times, only the last one is recorded
PauseNs
PauseEnd

/ / GC number
NumGC

// The number of times the application forced GC
NumForcedGC

// GC total CPU resources occupied. Between 0 ~ 1
GCCPUFraction

// It's not being used, just ignore it
DebugGC

]

See the way

[

1 / / way
import "runtime"

var m runtime.MemStats
runtime.ReadMemStats( & m)
2 / / way
import _ "net/http/pprof"
import "net/http"
http.ListenAndServe("0.0.0.0:10001", nil)
// http://127.0.0.1:10001/debug/pprof/heap?debug=1

]

Let's find some random service to practice with.

VIRT for the Top viewing program is about 2.4G and 1.7G for RES.

The indicators observed by web pprof correspond to the meanings mentioned above.

[

# Sys = 1842916040 ~1.7G
# HeapSys = 1711013888 ~1.6G
# HeapInuse = 1237483520 ~1.18G
# HeapAlloc = 1195472528 ~1.14G
HeapInuse - HeapAlloc = 40M
# HeapIdle = 473530368 ~451M
# HeapReleased = 61063168 ~58.2M
HeapIdle - HeapReleased = 393M

# Alloc = 1195472528 ~1.14G
# TotalAlloc = 426616278424 ~397.3G

# Lookups = 0
# Mallocs = 2907819388 ~ 2.9 billion objects
# Frees = 2901808898 ~ 2.9 billion objects
# HeapObjects = 60104.9 ~ 6.01 million objects

# Stack = 33390592 / 33390592 ~31.8M / 31.8M
# MSpan = 13542744 / 19906560 ~12.9M / 18.9M
# MCache = 55552 / 65536
# BuckHashSys = 2371870
# GCSys = 69398992
# OtherSys = 6768602

]

conclusion