Docker could not stop or remove the solution to the container service problem

Preface

Today, some development students gave me feedback that a container service could not operate stop, rm (docker rm-f) and kill, which means that this container service could not be terminated ~

Operating steps

(1) Executing the delete command cannot delete the directory of docker:

# ll /var/lib/docker/containers | grep caf8ef20f3c1
# cd /var/lib/docker/containers 
# rm -rf caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8

At this time, we will receive such an error:

rm: Unable to delete "/var/lib/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d1ES34e44246f242292040f1ef2889d0cb8/secrets": Device or resource busy

Unable to delete "/var/lib/docker/containers/caf8ef20f3c1c78f03ES5844ee23abc1d7e44246f242292040f1ef28889d0cb8/shm": Device or resource busy

(2) From the above error report, we can see that "secrets" and "shm" share mounting, which makes it impossible to delete. First find the mounting location, then cancel the mounting, and then delete it:

# cat /proc/mounts |grep "docker" |grep "caf8ef20f3c1"

(3) Unmount:

# umount /data/sys/var/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/secrets
# umount /data/sys/var/docker/containers/caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8/shm

(4) Check again:

# cat /proc/mounts |grep "docker" |grep "caf8ef20f3c1" // There are no more 

(5) Now delete the directory of docker:

# cd /var/lib/docker/containers 
# rm -rf caf8ef20f3c1c78f03a5844ee23abc1d7e44246f242292040f1ef288899d0cb8

(6) Delete the container service

Now we use rm or kill to delete the container service:

# docker rm -f caf8ef20f3c1c

Or

# docker kill --signal=SIGINT caf8ef20f3c1

If hang is stuck after the above command is run, restart the docker service:

# systemctl restart docker

After completing the above steps, the problems encountered can be basically solved ~

Added: Docker container can not be stop or kill problem

Problem process

An mysql container in an environment cannot be used by stop or kill or rm

sudo docker ps grep mysql View this container

7844250860f8 mysql:5.7.22  "/.r/r docker-entr..."  41 minutes ago  Up 8 minutes  r-dlrel-mysql-1-66df8f33

After processing with commands such as docker stop/docker kill/docker rm-f, the container automatically restarts

Look at the container immediately. The running time is Up Less than a second, indicating that the container is started immediately

7844250860f8 mysql:5.7.22  "/.r/r docker-entr..."  42 minutes ago Up Less than a second  r-dlrel-mysql-1-66df8f33

kill the physical process corresponding to this container is still automatically restarted

Obtain the physical process mode: 1. The State. Pid field in docker inspect is the physical process ID; 2. ps Command

Look at the container restart policy, the policy is no, that is, it will not restart automatically

If you need to update the restart policy for a running container, you can use this command: docker update restart=no my-container

# cat /proc/mounts |grep "docker" |grep "caf8ef20f3c1"

0

Magic problem solving among programmers

Have you ever had this scene:

When I walked to my colleagues and made the problem clear, I suddenly realized it.

The problem is very simple, but the program is running out of problem, and then find a colleague to help check the basic configuration, and I have an epiphany.

This time, I belong to the first type. As soon as I finished the problem, I immediately remembered: Wipe, it is the container orchestration tool Rancher that is scheduling, and the container will restart automatically after hanging up.

Log in to rancher1 to see, sure enough, "Oolong" problem. Although this is not a problem, Docker does have the problem that stop can't, and there are many materials.