Principle and application of Docker container memory monitoring
- 2020-05-17 07:06:53
- OfStack
Docker container memory monitoring
linux memory monitoring
To understand how docker container memory is computed, first understand the concepts of memory in linux.
Use the free command to see the current memory usage.
[root@localhost ~]$ free
total used free shared buffers cached
Mem: 264420684 213853512 50567172 71822688 2095364 175733516
-/+ buffers/cache: 36024632 228396052
Swap: 16777212 1277964 15499248
Here are a few ideas:
mem: physical memory swap: virtual memory. That is, data that can be stored on a hard disk shared: Shared memory. It's in physical memory. buffers: for storing data to be exported to disk (block device) cached: stores data read from diskYou can see it here.
For the sake of illustration, I made a corresponding result for free.
[root@localhost ~]$ free
total used free shared buffers cached
Mem: total_mem used_mem free_mem shared_mem buffer cache
-/+ buffers/cache: real_used real_free
Swap: total_swap used_swap free_swap
| 名称 | 说明 |
|---|---|
| total_mem | 物理内存总量 |
| used_mem | 已使用的物理内存量 |
| free_mem | 空闲的物理内存量 |
| shared_mem | 共享内存量 |
| buffer | buffer所占内存量 |
| cache | cache所占内存量 |
| real_used | 实际使用的内存量 |
| real_free | 实际空闲的内存量 |
| total_swap | swap总量 |
| used_swap | 已使用的swap |
| free_swap | 空闲的swap |
Generally speaking, buffer and cache are memory that can be reused, so when calculating free memory, they are removed.
So here are some equations:
real_used = used_mem - buffer - cache
real_free = free_mem + buffer + cache
total_mem = used_mem + free_mem
With that in mind, let's look at the data source for free. The data source is actually from the /proc/memeinfo file.
[root@localhost ~]$ cat /proc/meminfo
MemTotal: 264420684 kB
MemFree: 50566436 kB
Buffers: 2095356 kB
Cached: 175732644 kB
SwapCached: 123688 kB
Active: 165515340 kB
Inactive: 37004224 kB
Active(anon): 92066880 kB
Inactive(anon): 4455076 kB
Active(file): 73448460 kB
Inactive(file): 32549148 kB
Unevictable: 362900 kB
Mlocked: 74696 kB
SwapTotal: 16777212 kB
SwapFree: 15499248 kB
Dirty: 2860 kB
Writeback: 0 kB
AnonPages: 24932928 kB
Mapped: 58165040 kB
Shmem: 71822688 kB
Slab: 8374496 kB
SReclaimable: 8163096 kB
SUnreclaim: 211400 kB
KernelStack: 45824 kB
PageTables: 606296 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 148987552 kB
Committed_AS: 114755628 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 772092 kB
VmallocChunk: 34225428328 kB
HardwareCorrupted: 0 kB
AnonHugePages: 22083584 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7168 kB
DirectMap2M: 2015232 kB
DirectMap1G: 266338304 kB
docker
Having said linux's memory, let's take a look at docker's memory monitoring. docker itself provides a means of memory monitoring, that is, container memory can be monitored via docker stats.
This method is actually calculated by evaluating the relevant data in cgroup.
cgroup
The memory subsystem in cgroup provides the following files for hierarchy.
[root@localhost ~]$ ll /cgroup/memory/docker/53a11f13c08099dd6d21030dd2ddade54d5cdd7ae7e9e68f5ba055ad28498b6f/
The total amount 0
--w--w--w- 1 root root 0 2 month 22 12:51 cgroup.event_control
-rw-r--r-- 1 root root 0 5 month 25 17:07 cgroup.procs
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.failcnt
--w------- 1 root root 0 2 month 22 12:51 memory.force_empty
-rw-r--r-- 1 root root 0 3 month 30 17:06 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 3 month 30 17:06 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 2 month 22 12:51 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.move_charge_at_immigrate
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.oom_control
-rw-r--r-- 1 root root 0 3 month 30 17:06 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 2 month 22 12:51 memory.stat
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.swappiness
-r--r--r-- 1 root root 0 2 month 22 12:51 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 2 month 22 12:51 memory.use_hierarchy
-rw-r--r-- 1 root root 0 2 month 22 12:51 notify_on_release
-rw-r--r-- 1 root root 0 2 month 22 12:51 tasks
The specific meaning of these documents can be found in the relevant information cgroup memory.
Here are a few that are related to docker monitoring.
| 文件名 | 说明 |
|---|---|
| memory.usage_in_bytes | 已使用的内存量(包含cache和buffer)(字节)，相当于linux的used_meme |
| memory.limit_in_bytes | 限制的内存总量(字节)，相当于linux的total_mem |
| memory.failcnt | 申请内存失败次数计数 |
| memory.memsw.usage_in_bytes | 已使用的内存和swap(字节) |
| memory.memsw.limit_in_bytes | 限制的内存和swap容量(字节) |
| memory.memsw.failcnt | 申请内存和swap失败次数计数 |
| memory.stat | 内存相关状态 |
Here is a sample of a container.
[root@localhost 53a11f13c08099dd6d21030dd2ddade54d5cdd7ae7e9e68f5ba055ad28498b6f]$ cat memory.usage_in_bytes
135021858816
[root@localhost 53a11f13c08099dd6d21030dd2ddade54d5cdd7ae7e9e68f5ba055ad28498b6f]$ cat memory.memsw.usage_in_bytes
135679291392
[root@localhost 53a11f13c08099dd6d21030dd2ddade54d5cdd7ae7e9e68f5ba055ad28498b6f]$ cat memory.stat
cache 134325506048
rss 695980032
mapped_file 16155119616
pgpgin 21654116032
pgpgout 21705492352
swap 655171584
inactive_anon 4218880
active_anon 74202603520
inactive_file 8365199360
active_file 52449439744
unevictable 0
hierarchical_memory_limit 137438953472
hierarchical_memsw_limit 274877906944
total_cache 134325506048
total_rss 695980032
total_mapped_file 16155119616
total_pgpgin 21654116032
total_pgpgout 21705492352
total_swap 655171584
total_inactive_anon 4218880
total_active_anon 74202603520
total_inactive_file 8365199360
total_active_file 52449439744
total_unevictable 0
memory.stat
memory.stat contains the most abundant
| 统计 | 描述 |
|---|---|
| cache | 页缓存，包括 tmpfs（shmem），单位为字节 |
| rss | 匿名和 swap 缓存，不包括 tmpfs（shmem），单位为字节 |
| mapped_file | memory-mapped 映射的文件大小，包括 tmpfs（shmem），单位为字节 |
| pgpgin | 存入内存中的页数 |
| pgpgout | 从内存中读出的页数 |
| swap | swap 用量，单位为字节 |
| active_anon | 在活跃的最近最少使用（least-recently-used，LRU）列表中的匿名和 swap 缓存，包括 tmpfs（shmem），单位为字节 |
| inactive_anon | 不活跃的 LRU 列表中的匿名和 swap 缓存，包括 tmpfs（shmem），单位为字节 |
| active_file | 活跃 LRU 列表中的 file-backed 内存，以字节为单位 |
| inactive_file | 不活跃 LRU 列表中的 file-backed 内存，以字节为单位 |
| unevictable | 无法再生的内存，以字节为单位 |
| hierarchical_memory_limit | 包含 memory cgroup 的层级的内存限制，单位为字节 |
| hierarchical_memsw_limit | 包含 memory cgroup 的层级的内存加 swap 限制，单位为字节 |
active_anon + inactive_anon = anonymous memory + file cache for tmpfs + swap cache
active_file + inactive_file = cache - size of tmpfs
docker native memory monitoring
Take docker's native docker stats. The implementation can be seen in libcontainer. It divides the memory monitoring of the container into cache,usage,swap usage,kernel usage,kernel tcp usage.
Where cache is obtained from memory.stat is obtained from cache.
usage calculates usage by dividing memory.usage_in_bytes and memory.limit_in_bytes. One disadvantage of this 1 method is that it is not detailed enough to distinguish the cache part, which cannot truly reflect the memory utilization. Because 1 generally refers to cache as a reusable part of memory, 1 generally counts it as a usable part.
Can consider the improvement of the calculation method
The improved approach excludes the cache calculation from the memory usage statistics. It is similar to the calculation of real_used in linux in which buffer and cache are excluded.
cache does not directly apply memory. cache in stat, because it includes tmpfs, and tmpfs is the part of memory that is actually used.
tmpfs, share memory, Shared memory
Because it's in memory.stat
active_file + inactive_file = cache - size of tmpfs
Therefore, the actual amount of memory used can be calculated as
real_used = memory.usage_in_bytes - (rss + active_file + inactive_file)
Thank you for reading, I hope to help you, thank you for your support of this site!