Detailed Explanation of top Command Output in Linux

Preface

I believe everyone has used the top command under Linux. Since I contacted Linux, I have used top to view the CPU and MEM rankings of the process. But I don't know the other output of top command. What do these indicators represent and under what circumstances should I pay attention to? And what is the source data of the output result of top command, and how is it a calculation principle?

Demonstration environment

# uname -a
Linux VM_1_11_centos 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

top Command

top command is a common performance analysis tool under Linux, which can display the resource usage of the system and the resource usage of various processes in real time (the default is 3s refresh once), similar to the task manager of Windows.

top - 11:00:54 up 54 days, 23:35, 6 users, load average: 16.32, 18.75, 21.04
Tasks: 209 total,  3 running, 205 sleeping,  0 stopped,  1 zombie
%Cpu(s): 29.7 us, 18.9 sy, 0.0 ni, 49.3 id, 1.7 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 32781216 total, 1506220 free, 6525496 used, 24749500 buff/cache
KiB Swap:    0 total,    0 free,    0 used. 25607592 avail Mem 

 PID USER   PR NI  VIRT  RES  SHR S %CPU %MEM   TIME+ COMMAND                                                                                 
root   20  0  15.6g 461676  4704 R 198.0 1.4 11:15.26 python                                                                                  
root   20  0 9725596 240028  4672 R 113.0 0.7  7:48.49 python                                                                                  
root   20  0 6878028 143196  4720 S 82.4 0.4  1:35.03 python

The data in line 1 is equivalent to the output of uptime command. 11:00:54 is the current time, up 54 days, 23:55 is the time when the system has been running, 6 users indicates that there are currently 6 users logging in, and load average: 16.32, 18.75 and 21.04 indicate the average load of the system in 1 minute, 5 minutes and 15 minutes respectively.

Average load

Average load represents the average number of active processes, including the number of processes in running, the number of processes ready for running, and the number of processes in uninterrupted sleep. If the average load number is just equal to the number of CPU cores, it proves that each core can be well utilized. If the average load number is greater than the number of cores, it proves that the system is in an overloaded state, which is usually considered to be seriously overloaded and needs attention. It is also necessary to look at the load trend in combination with the 1-minute average load, 5-minute average load and 15-minute average load. If the 1-minute load is relatively high and the 5-minute and 15-minute average loads are relatively low, it means that it is instantaneous and needs to be observed. If all three values are high, it is necessary to pay attention to whether a process is consuming CPU crazily or has frequent IO operations, or it may be caused by too many processes running in the system and frequent process switching. For example, the above demonstration environment is an 8-core centos machine, which proves that the system is running under overload for a long time.

Tasks: 214 total, 4 running, 209 sleeping, 0 stopped, 1 zombie

The Tasks information in Line 2 shows the overall number of processes and status information that the system is running. 214 total indicates that the system now has 214 user processes, 4 running indicates that 4 processes are in running state, 209 sleeping indicates that 209 processes are in sleeping state, 0 stopped indicates that 0 processes are in stopped state, and 1 zombie indicates that there is a zombie process.

Zombie process

When the child process ends, the parent process does not call wait ()/waitpid () to wait for the child process to end, then a zombie process will be generated. The reason is that the child process does not really exit at the end, but leaves the data structure of a zombie process in the system process table, waiting for the parent process to clean up. If the parent process has exited, the init process will replace the parent process for processing (corpse collection). It can be seen that if the parent process does not act and does not exit, there will be a large number of zombie processes, and each zombie process will occupy one position in the process table (slot). If there are too many zombie processes, the system will not be able to create new processes, because the capacity of the process table is limited. Therefore, when zombie is too large, we need to pay attention to it. The S column in the process details below represents the running state of the process, and Z indicates that the process is a zombie process.

Ways to eliminate zombie process:

1. Find the parent process of the zombie process pid (pstress can show the parent-child relationship of the process), kill-9 pid, init will automatically clean up the zombie process after the parent process exits. (Note that kill-9 does not kill zombie processes.)

2. Restart the system.

%Cpu(s): 31.9 us, 30.3 sy, 0.0 ni, 37.0 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st

The% Cpu (s) in Line 3 represents the overall CPU usage.

us user represents the CPU time ratio of user mode sy system represents the CPU time ratio of the core state ni nice represents the proportion of CPU time to run a low priority process id idle represents the proportion of idle CPU time wa iowait represents the proportion of CPU time waiting in IO hi hard interrupt represents the CPU time ratio for processing hard interrupts si soft interrupt represents the CPU time ratio for processing soft interrupts st steal represents the proportion of CPU time spent by other virtual machines when the current system is running in the virtual machine.

Therefore, the overall CPU utilization rate = 1-id. When us is very high, it is proved that CPU time is mainly consumed in user code, and user code needs to be optimized. When sy is high, it means that CPU time is consumed in the kernel, which is either frequent system calls or frequent CPU switching (process switching/thread switching). When wa is very high, it means that there are frequent IO operations in the process, which may be disk IO or network IO. When si is very high, it means that CPU time is consumed in processing soft interrupts, and network sending and receiving packets will trigger system soft interrupts, so a large number of network packets will lead to frequent triggering of soft interrupts, and typical SYN Floor will lead to high si.

KiB Mem : 32781216 total,  663440 free, 7354900 used, 24762876 buff/cache
KiB Swap:    0 total,    0 free,    0 used. 24771700 avail Mem

Lines 4 and 5 show the system memory usage. The unit is KiB. totol represents total memory, free represents unused content, and used is used memory. buff represents the memory used to read and write the disk cache, and cache represents the memory used to read and write the file cache. avail represents the available application memory.

The principle of Swap is to use one block of disk space or one local file as memory. Swap total represents the total amount of swap available, swap free represents the remaining, and used represents the used. These three values are all 0, indicating that the system has turned off swap function. Since the demonstration environment is a virtual machine, virtual machine 1 generally turns off swap function.

Line 6 begins with the specific state of each process:

PID USER   PR NI  VIRT  RES  SHR S %CPU %MEM   TIME+ COMMAND 

PID Process ID The user name of the USER process owner, such as root PR Process Scheduling Priority NI process nice value (priority), the smaller the value, the higher the priority Virtual Memory Used by VIRT Processes Physical memory used by RES process (excluding shared memory) Shared Memory Used by SHR Processes Proportion of CPU used by CPU process Percentage of memory used by MEM process Total CPU time since the TIME process started Startup command for COMMAND process (only binary is displayed by default, top-c can display command line and startup parameters)

Principle of calculation

Before introducing the calculation principle of each index of top command, it is necessary to introduce the proc file system under Linux first, because the data of top command comes from proc file system. proc file system is a virtual file system, which is a communication mode between Linux kernel and users. Linux kernel will tell users the current status information of the kernel through proc file system, and users can also set some behaviors of the kernel by writing proc. Unlike ordinary files, these proc files are created dynamically and modified dynamically because the state of the kernel changes all the time.

The CPU indicators shown in top are all derived from/proc/stat file information:

# cat /proc/stat 
cpu 1151829380 20277 540128095 1909004524 21051740 0 10957596 0 0 0
cpu0 143829475 3918 67658924 235696976 5168514 0 1475030 0 0 0
cpu1 144407338 1966 67616825 236756510 3969110 0 1392212 0 0 0
cpu2 144531920 2287 67567520 238021699 2713175 0 1363460 0 0 0
cpu3 143288938 2366 67474485 239715220 2223739 0 1356698 0 0 0
cpu4 143975390 3159 67394206 239494900 1948424 0 1343261 0 0 0
cpu5 144130685 2212 67538520 239431294 1780756 0 1349882 0 0 0
cpu6 144009592 2175 67536945 239683876 1668203 0 1340087 0 0 0
cpu7 143656038 2193 67340668 240204045 1579816 0 1336963 0 0 0

Line 1 represents the total CPU information, followed by the details of one CPU.

But what are these specific following columns? We can find out through man proc:

user  (1) Time spent in user mode.

nice  (2) Time spent in user mode with low priority (nice).

system (3) Time spent in system mode.

idle  (4) Time spent in the idle task. This value should be USER_HZ times the second entry in the
      /proc/uptime pseudo-file.
iowait (since Linux 2.5.41)

     (5) Time waiting for I/O to complete.

irq (since Linux 2.6.0-test4)
     (6) Time servicing interrupts.

softirq (since Linux 2.6.0-test4)
     (7) Time servicing softirqs.

steal (since Linux 2.6.11)
     (8) Stolen time, which is the time spent in other operating systems when running in a virtual In fact, in fact, the 
         ized environment

guest (since Linux 2.6.24)
     (9) Time spent running a virtual CPU for guest operating systems under the control of the Linux kernel.

guest_nice (since Linux 2.6.33)
      (10) Time spent running a niced guest (virtual CPU for guest operating systems under the con In fact, in fact, the 
         trol of the Linux kernel).

That is to say, from the second column, there are user, nice, system, idle, iowait, irq (hard interrupt), softirq (soft interrupt), steal, guest, guest_nice. The unit is usually 10ms. So how is the proportion in top calculated?

Since CPU time is an accumulated value, we require a time period difference to reflect the current CPU situation, and top defaults to 3s. For example, now take 1 user value user1 and the current 1 total CPU time total1

Where total is equal to the sum of the above items, that is, total=user+nice+system+idle+iowait+irq+softirq+steal+guest+guest_nice. After 3 seconds, go to one user value user2 and one total total2.

Then the average cpu ratio of user in these 3 seconds is equal to ((user2-user1)/(total2-total1))/3 * 100%. In addition, the calculation method of each specific CPU is the same.

top memory-related metrics directly read the corresponding fields in the/proc/meminfo file:

# cat /proc/meminfo 
MemTotal:    32781216 kB
MemFree:     1043556 kB
MemAvailable:  25108920 kB
Buffers:     427516 kB
Cached:     22084612 kB
SwapCached:      0 kB
Active:     18640888 kB
Inactive:    10534920 kB
Active(anon):  6664480 kB
Inactive(anon):   412 kB
Active(file):  11976408 kB
Inactive(file): 10534508 kB
Unevictable:      4 kB
Mlocked:        4 kB
SwapTotal:       0 kB
SwapFree:       0 kB
Dirty:       1092 kB
Writeback:       0 kB
AnonPages:    6663764 kB
Mapped:      347808 kB
Shmem:       1212 kB
Slab:      2201292 kB
SReclaimable:  1957344 kB
SUnreclaim:    243948 kB
KernelStack:    73392 kB
PageTables:    57300 kB
NFS_Unstable:     0 kB
Bounce:        0 kB
WritebackTmp:     0 kB
CommitLimit:  16390608 kB
Committed_AS:  42170784 kB
VmallocTotal:  34359738367 kB
VmallocUsed:    61924 kB
VmallocChunk:  34359625048 kB
HardwareCorrupted:   0 kB
AnonHugePages:  364544 kB
HugePages_Total:    0
HugePages_Free:    0
HugePages_Rsvd:    0
HugePages_Surp:    0
Hugepagesize:    2048 kB
DirectMap4k:   376680 kB
DirectMap2M:  26886144 kB
DirectMap1G:   8388608 kB

Where total corresponds to MemTotal, free corresponds to MemFree, and avail corresponds to MemAailable.

Summarize

Beginning with the output of top command, this paper explains which outliers need our attention. Finally, it introduces the cpu calculation principle of top command and the data source of mem.