Detailed processing steps after the Linux server was hacked
- 2020-06-23 02:36:29
- OfStack
With the increasing popularity of open source products, as an Linux operation and maintenance engineer, it is very important to be able to clearly identify whether abnormal machines have been hacked. Based on my own work experience, I have sorted out several common machine hacking situations for reference
Background: The following is viewed in CentOS 6.9, similar to other Linux distributions
1. The intruder may delete the log information of the machine. You can check whether the log information still exists or is cleared.
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="98eaf7f7ecd8f0f4f5fbfdf6aea1f6ab">[email protected]</a> ~]# ll -h /var/log/*
-rw-------. 1 root root 2.6K Jul 7 18:31 /var/log/anaconda.ifcfg.log
-rw-------. 1 root root 23K Jul 7 18:31 /var/log/anaconda.log
-rw-------. 1 root root 26K Jul 7 18:31 /var/log/anaconda.program.log
-rw-------. 1 root root 63K Jul 7 18:31 /var/log/anaconda.storage.log
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="493b26263d092125242a2c277f70277a">[email protected]</a> ~]# du -sh /var/log/*
8.0K /var/log/anaconda
4.0K /var/log/anaconda.ifcfg.log
24K /var/log/anaconda.log
28K /var/log/anaconda.program.log
64K /var/log/anaconda.storage.log2. The intruder may create a new file to store user name and password. You can view /etc/passwd and /etc/shadow files.
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="80f2efeff4c0e8ecede3e5eeb6b9eeb3">[email protected]</a> ~]# ll /etc/pass*
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd-
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="c8baa7a7bc88a0a4a5abada6fef1a6fb">[email protected]</a> ~]# ll /etc/sha*
----------. 1 root root 816 Sep 15 11:36 /etc/shadow
----------. 1 root root 718 Sep 15 11:36 /etc/shadow-3. The intruder may modify the user name and password file. The /etc/passwd and /etc/shadow files can be viewed for identification.
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="9ceef3f3e8dcf4f0f1fff9f2aaa5f2af">[email protected]</a> ~]# more /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="20524f4f5460484c4d43454e16194e13">[email protected]</a> ~]# more /etc/shadow
root:*LOCK*:14600::::::
bin:*:17246:0:99999:7:::
daemon:*:17246:0:99999:7:::4. Check the recent successful login event of the machine and the last unsuccessful login event, corresponding to the log "/var/log/lastlog", relevant command examples:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="4e3c21213a0e2622232d2b207877207d">[email protected]</a> ~]# lastlog
Username Port From Latest
root **Never logged in**
bin **Never logged in**
daemon **Never logged in**5. Check all users currently logged on the machine, corresponding to the log file "/var/run/utmp", and relevant command examples:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="c3b1acacb783abafaea0a6adf5faadf0">[email protected]</a> ~]# who
stone pts/0 2017-09-20 16:17 (X.X.X.X)
test01 pts/2 2017-09-20 16:47 (X.X.X.X)6. Check the users logged in since the machine was created, corresponding to the log file "/var/log/wtmp", and relevant command examples:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="ec9e838398ac8480818f8982dad582df">[email protected]</a> ~]# last
test01 pts/1 X.X.X.X Wed Sep 20 16:50 still logged in
test01 pts/2 X.X.X.X Wed Sep 20 16:47 - 16:49 (00:02)
stone pts/1 X.X.X.X Wed Sep 20 16:46 - 16:47 (00:01)
stone pts/0 X.X.X.X Wed Sep 20 16:17 still logged in7. Check the connection time (hours) of all users of the machine, corresponding to the log file "/var/log/wtmp", and relevant command examples:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="03716c6c77436b6f6e60666d353a6d30">[email protected]</a> ~]# ac -dp
stone 11.98
Sep 15 total 11.98
stone 67.06
Sep 18 total 67.06
stone 1.27
test01 0.24
Today total 1.508. If the machine is found to generate abnormal traffic, you can use the command "tcpdump" to grab the network packet and view the traffic situation or use the tool "iperf" to view the traffic situation
9. You can view /var/log/secure log file and try to find the information of the intruder.
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="82f0ededf6c2eaeeefe1e7ecb4bbecb1">[email protected]</a> ~]# cat /var/log/secure | grep -i "accepted password"
Sep 20 12:47:20 hlmcen69n3 sshd[37193]: Accepted password for stone from X.X.X.X port 15898 ssh2
Sep 20 16:17:47 hlmcen69n3 sshd[38206]: Accepted password for stone from X.X.X.X port 9140 ssh2
Sep 20 16:46:00 hlmcen69n3 sshd[38511]: Accepted password for stone from X.X.X.X port 2540 ssh2
Sep 20 16:47:16 hlmcen69n3 sshd[38605]: Accepted password for test01 from X.X.X.X port 10790 ssh2
Sep 20 16:50:04 hlmcen69n3 sshd[38652]: Accepted password for test01 from X.X.X.X port 28956 ssh210. Query the execution script file corresponding to the exception process
The a. top command looks at PID for the exception process
b. Find the executable file for the process in the virtual file system directory
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="285a47475c684044454b4d461e11461b">[email protected]</a> ~]# ll /proc/1850/ | grep -i exe
lrwxrwxrwx. 1 root root 0 Sep 15 12:31 exe -> /usr/bin/python
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="1d6f7272695d7571707e78732b24732e">[email protected]</a> ~]# ll /usr/bin/python
-rwxr-xr-x. 2 root root 9032 Aug 18 2016 /usr/bin/python11. If it is confirmed that the machine has been hacked and important files have been deleted, you can try to recover the deleted files
Note:
1 > When a process opens a file, it remains on disk as long as the process keeps the file open, even if it is deleted. This means that the process does not know that the file has been deleted, and it can still read and write to the file descriptor provided to it when it opens the file. Except for the process, the file is not visible because its corresponding directory inodes have been removed.
2 > The /proc directory contains various files that reflect the kernel and process trees. The /proc directory mounts a block mapped in memory, so these files and directories do not exist on disk, so when we read and write these files, we are actually retrieving information from memory. Most of the information related to lsof is stored in a directory named after the PID of the process, that is, /proc/1234 contains information about the process of PID 1234. Each process directory contains a variety of files that allow applications to easily understand the process's memory space, the list of file descriptors, symbolic links to files on disk, and other system information. The lsof program USES this and other information about the internal state of the kernel to produce its output. So lsof can display information such as the file descriptor of the process and the associated file name. That is, we can access the file descriptor of the process to find information about the file.
3 > When a file in the system is accidentally deleted, as long as it is being accessed by a process in the system, we can recover the file from the /proc directory via lsof.
Assuming that the /var/log/secure file has been deleted by the intruder, the method of trying to recover the /var/log/secure file can be referred to as follows:
a. View the /var/log/secure file and find that it is no longer available
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="20524f4f5460484c4d43454e16194e13">[email protected]</a> ~]# ll /var/log/secure
ls: cannot access /var/log/secure: No such file or directoryb. Use the lsof command to see if any processes are currently open /var/log/secure,
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="80f2efeff4c0e8ecede3e5eeb6b9eeb3">[email protected]</a> ~]# ll /etc/pass*
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd-
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="c8baa7a7bc88a0a4a5abada6fef1a6fb">[email protected]</a> ~]# ll /etc/sha*
----------. 1 root root 816 Sep 15 11:36 /etc/shadow
----------. 1 root root 718 Sep 15 11:36 /etc/shadow-c. From the above information you can see that the file descriptor for PID 1264 (rsyslogd) to open the file is 4. You can also see that /var/log/ secure has been marked as deleted. Therefore, we can see the corresponding information in /proc/1264/fd/4 (each numeric file under fd represents the file descriptor for the process), as follows:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="80f2efeff4c0e8ecede3e5eeb6b9eeb3">[email protected]</a> ~]# ll /etc/pass*
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd-
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="c8baa7a7bc88a0a4a5abada6fef1a6fb">[email protected]</a> ~]# ll /etc/sha*
----------. 1 root root 816 Sep 15 11:36 /etc/shadow
----------. 1 root root 718 Sep 15 11:36 /etc/shadow-d. As you can see from the above information, look at /proc/1264/fd/4 to get the data you want to recover. If the corresponding data can be viewed through the file descriptor, then I/O redirection can be used to redirect it to the file, such as:
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="80f2efeff4c0e8ecede3e5eeb6b9eeb3">[email protected]</a> ~]# ll /etc/pass*
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd
-rw-r--r--. 1 root root 1373 Sep 15 11:36 /etc/passwd-
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="c8baa7a7bc88a0a4a5abada6fef1a6fb">[email protected]</a> ~]# ll /etc/sha*
----------. 1 root root 816 Sep 15 11:36 /etc/shadow
----------. 1 root root 718 Sep 15 11:36 /etc/shadow-e. Looking at /var/log/secure again, the file already exists. This method of recovering deleted files is useful for many applications, especially log files and databases.
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="63110c0c17230b0f0e00060d555a0d50">[email protected]</a> ~]# ll /var/log/secure
-rw-r--r--. 1 root root 3173904 Sep 20 17:24 /var/log/secure
[<a href="/cdn-cgi/l/email-protection" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" data-cfemail="ec9e838398ac8480818f8982dad582df">[email protected]</a> ~]# head /var/log/secure
Sep 17 03:28:15 hlmcen69n3 sshd[13288]: reverse mapping checking getaddrinfo for 137-64-15-51.rev.cloud.scaleway.com [51.15.64.137] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 17 03:28:15 hlmcen69n3 unix_chkpwd[13290]: password check failed for user (root)
Sep 17 03:28:15 hlmcen69n3 sshd[13288]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.15.64.137 user=root
Sep 17 03:28:17 hlmcen69n3 sshd[13288]: Failed password for root from 51.15.64.137 port 59498 ssh2
Sep 17 03:28:18 hlmcen69n3 sshd[13289]: Received disconnect from 51.15.64.137: 11: Bye Bye
Sep 17 03:28:22 hlmcen69n3 sshd[13291]: reverse mapping checking getaddrinfo for 137-64-15-51.rev.cloud.scaleway.com [51.15.64.137] failed - POSSIBLE BREAK-IN ATTEMPT!
Sep 17 03:28:22 hlmcen69n3 unix_chkpwd[13293]: password check failed for user (root)
Sep 17 03:28:22 hlmcen69n3 sshd[13291]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.15.64.137 user=root
Sep 17 03:28:24 hlmcen69n3 sshd[13291]: Failed password for root from 51.15.64.137 port 37722 ssh2
Sep 17 03:28:25 hlmcen69n3 sshd[13292]: Received disconnect from 51.15.64.137: 11: Bye Byeconclusion