Health detection mechanism of Docker

The simplest health check for a container is a process-level health check to see if the process is alive. Docker Daemon automatically monitors PID1 processes in the container, and if the docker run command indicates restart policy, the finished container can be automatically restarted according to the policy. In many real-world scenarios, the use of a process-level health check mechanism alone is insufficient. For example, if a container process is still running but cannot continue to respond to user requests due to the application deadlock, such problems cannot be detected by process monitoring.

In case the container fails to start automatically after power failure or abnormal shutdown, we can add

--restart=always 

For example,

[root@aliyun ~]# docker run --restart=always -d --name blog -d -v /www:/www -v /wwwlogs:/var/log/wwwlogs -p 65423:65422 -p 80:80 -p 443:443 677
7714a84063ee6d405c80b891254bba0e5930f5d271c5ad76cfd6e2f0058d8056

This allows the container to be restarted automatically, but sometimes the application enters a deadlock state, or an endless loop state, the application process does not exit, but the container is no longer available. Before 1.12, Docker will not detect this state of the container and will not reschedule, resulting in the possibility that some containers are still accepting user requests when they are no longer available.

Since 1.12, Docker provides HEALTHCHECK instruction, which specifies 1 line of command to determine whether the service state of the container main process is still normal, so as to reflect the actual state of the container.

When the HEALTHCHECK instruction is specified on a mirror and used to start the container, it will start at starting, change to healthy after the HEALTHCHECK instruction is checked successfully, and change to unhealthy if it fails for a fixed number of times in a row.

HEALTHCHECK supports the following options:

� interval = < interval > : The default interval between two health checks is 30 seconds; � timeout = < The length > : The health check command runs out of time. If the health check runs out of time, the health check will be considered a failure. The default is 30 seconds. � retries = < The number of > : After specified number of consecutive failures, the container state is treated as unhealthy, with the default of 3 times. Like CMD, ENTRYPOINT 1, HEALTHCHECK can only appear once, if more than one is written, only the last one takes effect.

The commands after HEALTHCHECK [option] CMD, format and ENTRYPOINT 1, are divided into shell format and exec format. The return value of the command determines the success of the health check: 0: success; 1: Failure; 2: Reserve, do not use this value.

Let's take a look at this dockerfile file

FROM centos
LABEL maintainer "awen Email: <hi@awen.me>"
WORKDIR /opt/

COPY CentOS7-Base-163.repo /etc/yum.repos.d/CentOS-Base.repo
COPY nginx /etc/init.d/nginx

ENV NGINX_V=1.13.5 \
  OPENSSL_V=1.0.2l \
  PCRE_V=8.41 \
  ZLIB_V=1.2.11 

RUN yum -y update \
  && yum -y install openssh-server openssl gcc gcc-c++ pcre-devel openssl-devel zlib-devel wget make perl tar net-tools \
  && wget -c -4 https://nginx.org/download/nginx-$NGINX_V.tar.gz \
  && wget -c -4 https://www.openssl.org/source/openssl-$OPENSSL_V.tar.gz \
  && wget -c -4 ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-$PCRE_V.tar.gz \
  && wget -c -4 http://zlib.net/zlib-$ZLIB_V.tar.gz \
  && groupadd -r www && useradd -r -g www www \
  && tar zxvf zlib-$ZLIB_V.tar.gz \
  && cd zlib-$ZLIB_V \
  && ./configure \
  && make \
  && make install \
  && cd /opt \
  && tar zxvf pcre-$PCRE_V.tar.gz \
  && cd pcre-$PCRE_V \
  && ./configure \
  && make \
  && make install \
  && cd /opt \
  && tar zxvf openssl-$OPENSSL_V.tar.gz \
  && tar zxvf nginx-$NGINX_V.tar.gz \
  && cd nginx-$NGINX_V \
  && ./configure --prefix=/usr/local/nginx --user=www --group=www --with-pcre=/opt/pcre-$PCRE_V --with-http_ssl_module --with-zlib=/opt/zlib-$ZLIB_V --with-openssl=/opt/openssl-$OPENSSL_V --with-http_v2_module --with-http_ssl_module \
  && make \
  && make install \
  && rm -rf /opt/* \
  && mkdir -p /usr/local/nginx/ssl \
  && mkdir -p /usr/local/nginx/conf/vhost \
  && mkdir -p /var/log/wwwlogs/ \
  && mkdir -p /www/ \
  && ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N '' \
  && ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N '' \
  && ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N '' \
  && ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N '' \
  && echo "RSAAuthentication yes" >> /etc/ssh/sshd_config \
  && echo "PubkeyAuthentication yes" >> /etc/ssh/sshd_config  \
  && sed -i "s/PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config \
  && sed -i "s/UsePAM yes/UsePAM no/g" /etc/ssh/sshd_config \
  && sed -i "s/#Port 22/Port 65422/g" /etc/ssh/sshd_config \
  && yum clean all \
  && mkdir /var/run/sshd \
  && chmod +x /etc/init.d/nginx \
  && rm -rf /root/*.cfg \
  && echo "Asia/Shanghai" > /etc/localtime


COPY ssl/* /usr/local/nginx/ssl/ 
COPY vhost/* /usr/local/nginx/conf/vhost/
COPY nginx.conf /usr/local/nginx/conf/
COPY ssh/* /root/.ssh/


VOLUME ["/www","/var/log/wwwlogs","/usr/local/nginx/ssl","/usr/local/nginx/conf/vhost"]

EXPOSE 65422 80 443

HEALTHCHECK CMD curl -fs http://localhost/ || exit 1
ENTRYPOINT /etc/init.d/nginx start && chown -R www:www /var/log/wwwlogs/ && /usr/sbin/sshd -D

Among them

HEALTHCHECK CMD curl -fs http://localhost/ || exit 1

It is the added health monitoring configuration, and then it is compiled and launched, and when you look at the process you will find that its state is starting

[root@aliyun ~]# docker ps -a
CONTAINER ID    IMAGE        COMMAND         CREATED       STATUS              PORTS                                NAMES
7714a84063ee    677         "/bin/sh -c '/etc/ini"  3 seconds ago    Up 2 seconds (health: starting)  0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:65423->65422/tcp  blog

If you wait a moment, you'll see that its status is healthy

[root@aliyun ~]# docker ps -a
CONTAINER ID    IMAGE        COMMAND         CREATED       STATUS            PORTS                                NAMES
7714a84063ee    677         "/bin/sh -c '/etc/ini"  About a minute ago  Up About a minute (healthy)  0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:65423->65422/tcp  blog

We can check the status of the last 3 times via inspect

[root@aliyun ~]# docker inspect --format '{{json .State.Health}}' blog | python -m json.tool
{
  "FailingStreak": 0,
  "Log": [
    {
      "End": "2017-10-11T11:15:27.516562686+08:00",
      "ExitCode": 0,
      "Output": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",
      "Start": "2017-10-11T11:15:27.470554485+08:00"
    },
    {
      "End": "2017-10-11T11:15:57.563377729+08:00",
      "ExitCode": 0,
      "Output": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",
      "Start": "2017-10-11T11:15:57.516690754+08:00"
    },
    {
      "End": "2017-10-11T11:16:27.609685416+08:00",
      "ExitCode": 0,
      "Output": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",
      "Start": "2017-10-11T11:16:27.563533362+08:00"
    },
    {
      "End": "2017-10-11T11:16:57.654441173+08:00",
      "ExitCode": 0,
      "Output": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",
      "Start": "2017-10-11T11:16:57.609810588+08:00"
    },
    {
      "End": "2017-10-11T11:17:27.701113019+08:00",
      "ExitCode": 0,
      "Output": "<html>\r\n<head><title>301 Moved Permanently</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n",
      "Start": "2017-10-11T11:17:27.654580727+08:00"
    }
  ],
  "Status": "healthy"
}

If the health check fails more times in a row than the number of retries, the status is changed to (unhealthy).