How does Linux handle the problem that files have been deleted but space is not released

Background of the problem

The server monitoring system of a business system sends an early warning notification, The utilization rate of disk space has reached 90%, and then I logged in to the server to search for a relatively large log file, all of which were deleted (the pit was buried here), and the disk space was released by 1. At that time, I was also negligent, and I didn't confirm whether the space of the file size found and deleted had been completely released. A few days later, the server was warned again, wondering how the log grew so fast. After investigation, it was found that after the last operation deleted the file, there was a large file space that was not released.

Problem Restoration and Solution

Find data files that take up more space

# View disk space usage 
$ df -h
# Query first /tmp Files that take up a large amount of space under the directory 
$ du -sh /tmp/*|sort -nr|head -3
# In the query /home Files that take up a large amount of space under the directory 
$ du -sh /home/*|sort -nr|head -3
#  After finding the file, delete it. After deleting, use  df -h Check to see if it has been released 

Explain why when you are looking for files, you first look for files in the directory /tmp/*

Linux System Deletion Policy: Linux has no recycle bin function, so the service will first move the files to the system/tmp directory, and then regularly clear the data in the/tmp directory.

Many servers did not partition the/tmp separately when installing the system, so it is possible that the data in the/tmp directory occupies a large part of the space, so you can clear the files in the/tmp directory first to release the space.

This delete file does not free space and occurs when one dubbo service log file under the/home directory space is deleted.

Why Delete File Does Not Free Space

1 Under normal circumstances, the space will not be released after deleting the file, but it will still appear when the file is locked by the process or the process 1 writes data directly to the file. Understand the principle of file storage mechanism and storage structure under Linux to understand this problem.

The file exists in Linux system and is divided into two parts: pointer part and data part.

Pointer section: In the file system meta-data, this pointer is cleared from meta-data after we execute the rm command to delete the data. Data section: The data is stored directly on the disk. When the pointer is cleared from meta-data, the space occupied by the data section can be overwritten and new contents can be written.

The reason why the space is not released after deleting the dubbo log file is that the dubbo process is still writing data directly to this file at 1. When deleting the file, the pointer is not cleared from meta-data, so the log file still occupies space.

How do I find such a file

You can get a list of files that have been deleted but are still occupied by the program through the lsof command:

lsof | grep delete

How do I free up such space

There are many ways to free up space to solve this type of problem: restart the occupied process, restart the operating system, and pass the command. It is most convenient to use the first two methods in non-production environment, but for production environment, try to use the command method. In fact, the command is also very simple:

echo " " >/home/dubbo/log/xxx.log

In this way, the occupied disk space will be released, and the process will not be affected to continue executing.