Linux Disk Space Free Problem Sorting

The/partition usage of one server in IDC is full! Has reached 100%! After checking, it was found that a file was too large (80G), so rm-f decisively deleted the file after confirming with relevant colleagues. However, it was found that after deleting the file, the disk space of the/partition was not released at all, and the utilization rate was still 100%! Why is this? ?

[root@linux-node1 ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/mapper/VolGroup00-LogVol00 58G 7.8G 47G 100% /tmpfs 1.9G 0 1.9G 0% /dev/shm/dev/vda1 190M 72M 108M 40% /boot

Cause analysis

In Linux system, deleting a file through rm or file manager only unlinks it from the directory structure of the file system (unlink), that is to say, only deletes the link between the file and the system directory structure; If the file is open at the time of deletion (1 process is using the file, the file is locked by the process, or Process 1 is writing data to the file, etc.), the process will still be able to read the file, that is to say, the file is not deleted in the state of reading, so the disk space will be occupied directly.

The storage of 1 file in the file system is divided into two parts: The data part and the pointer part, The pointer is located in meta-data of the file system, After the data is deleted, This pointer is cleared from meta-data, While the data part is stored on disk, After the pointer corresponding to the data is cleared from meta-data, The space occupied by the data part of the file can be overwritten and new contents can be written. The reason why after deleting files, The space hasn't been released yet, It is because some processes are still writing directly to this file at 1, which leads to the deletion of the file, but the pointer corresponding to the file has not been cleared from meta-data due to the process lock, and because the pointer has not been deleted, the system kernel thinks that the file has not been deleted, so it is not surprising that the query space has not been released through df command.

There are several solutions

1. Get the list of files that have been deleted but are still occupied by the application through the lsofgrep deleted command, and then kill drops the process that is still occupying the deleted files. It should be noted that if many processes are using the deleted files, it is a bit troublesome and risky to adopt the kill process in the first way. Because the kill process can force the system to recycle files allocated for use by truncating files in the proc file system. You must make sure that it will not affect the running process before you can use it. The application does not support this method well. When a file in use is truncated, it may cause unpredictable problems.

2. Or stop or restart the application using this deleted file, and let OS automatically reclaim disk space.

3. You can also restart the operating system, but this is not the best method. 4. To free the disk space occupied by the file, the best way is to empty the file online. In this way, not only can disk space be released immediately, but also the process can be guaranteed to continue writing logs to files.

How to empty files online (e.g./home/wangshibo. log):

# echo " " > /home/wangshibo. logb) # cat/dev/null > /home/wangshibo. logc) # > /home/wangshibo.log

There is also a phenomenon of disk space usage problem: obviously using df-h command to check that the disk space utilization rate is not high, and there is still a lot of free space, but when creating files or writing data, 1 directly reports an error and the disk is full: "no space left on device"!

1 This kind of problem is caused by the fact that the resource space after deleted deletion in the partition directory is not really released. The specific processing flow is as follows:

1. Check the disk usage under df-lh first, and find that Used under/data partition has used a lot of space, but the actual view does not take up so much space! 2. Locate the partition where the deleted files are located, such as/data partition 3. Look at all deleted files: lsof-n/data ES90deleted 4. Kill the delete process of these files and free up space: lsof-n/data deletedawk '{print $2}' ES100kill-95. Then run lsof-n/data grep delete and there should be no results. 6. Note: When the deleted process is just killed, df-h looks at the/data partition, and the used space of Used may be too large instantly, but with the killing of deleted process, resources are gradually released, and the used space of Used under the/data partition will gradually decrease and the available space of Avail will gradually increase)

Most file systems reserve 1 part of space for emergencies (such as full hard disk space), which can ensure that some key applications (such as databases) have some room when the hard disk is full, so as not to immediately crash, giving monitoring systems and administrators 1 time to notice. However, sometimes this part of the reserved hard disk space is a bit wasteful if it is not used.

In Linux system, 5% disk space is usually reserved by default on ext2, ext3 and ext4 file systems. For example, if the disk is 2TB, it means that 100GB space will be reserved, so will it seem a bit wasteful? You can change the 5% default setting by using the "tune2fs" command, such as reserving only 2% space. However, it is not recommended to set it to 0%, which is unsafe in real environment.

# df -TFilesystem Type 1K-blocks Used Available Use% Mounted on/dev/vda1 ext4 41151808 4962148 34076228 13% /devtmpfs devtmpfs 1931468 0 1931468 0% /devtmpfs tmpfs 1941204 0 1941204 0% /dev/shmtmpfs tmpfs 1941204 652 1940552 1% /runtmpfs tmpfs 1941204 0 1941204 0% /sys/fs/cgrouptmpfs tmpfs 388244 0 388244 0% /run/user/0[root@ss-server ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/vda1 40G 4.8G 33G 13% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 620K 1.9G 1% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgrouptmpfs 380M 0 380M 0% /run/user/0

For example, the above "/" partition is the ext4 file system, and the default system reserves 5% of the space, that is, 2G. You can now change the system reservation to 2% with the "tune2fs" command.

# tune2fs -m 2 /dev/vda1tune2fs 1.42.9 (28-Dec-2013)Setting reserved blocks percentage to 2% (209704 blocks)

After execution, it is found that the "/" partition frees up 1G of space, and then the system reserves 2% of the space.

[root@ss-server ~]# df -hFilesystem Size Used Avail Use% Mounted on/dev/vda1 40G 4.8G 34G 13% /devtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 620K 1.9G 1% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgrouptmpfs 380M 0 380M 0% /run/user/0