Detail the configuration of Docker network mode VLAN

  • 2020-05-17 07:06:58
  • OfStack

preface

As one of the most popular lightweight container technologies, Docker has many admirable features, such as mirror management for Docker. However, Docker also has many imperfections. The network aspect is the weak part of Docker. Therefore, it is necessary to have a deep understanding of Docker's network knowledge to meet higher network needs.

Docker network mode selection

At present, many articles have introduced the network model of Docker, but there are still many pitfalls and points to be paid attention to in the practical application

When Docker is applied to a production environment, there are several choices of network models

1. Native Bridge NAT mode

2. Linux Bridge VLAN mode

3. Use the network scheme of the third party

Native Bridge NAT mode

This is the native network mode of Docker, where the container of each host is in a separate subnet, and external access must be through the way of host port mapping. At the same time, container access between different hosts must also be through this way of host port mapping. In other words, the container on one host does not know about the container on the other host. Whether this method can be used for production or not, I was depressed at the beginning. At the same time, I wrote an article before about the high performance loss of NAT. Without resources to do a complete test, we didn't dare to use this network scheme in our original plan. However, in recent tests of my own, the performance of NAT is acceptable (QPS and Latency and VLAN are close to each other), as long as there is a suitable scheme to connect containers from different hosts. You can use Mesos+Marathon+Bamboo+HAProxy.

Linux Bridge VLAN mode

This is the reason why I wanted to make a decision when considering Docker network model 1.

1. I was afraid to use NAT at first

2. Other tools for the third meal are not quite mature

3. 1 the number of containers at the beginning cannot be too much (because the mode of VLAN is limited by the total number of VLAN, which can only be 4096 containers). If it is calculated as 10-16 containers per host, it can support 256 hosts, which is still acceptable

4. Each host needs a separate IP and can be interoperable

5, operation and maintenance management to be simple, after all, our operation and maintenance system or physical machine system

6, can achieve the host network and VLAN network isolation

The host is CentosOS 7.X series, and the network configuration of the host is as follows: two 1G network CARDS are attached to 1 by means of Bond, then 1 virtual network card is configured on VALN 1, and Docker0 Bridge of the container is configured on another VLAN 1

Network card configuration:

It is important to note that the bridge-utils, NetworkManager packages must be installed

steps

Configure two network CARDS, do not configure IP/GATEWAY and other network parameters, increase MASTER=bond0 As well as SLAVE=yes

Complete examples are:


TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=enp2s0f0
UUID=7f6fa8e9-0177-46a8-b8ea-55c2187bea11
DEVICE=enp2s0f0
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Added bond0 network configuration /etc/sysconfig/network-scripts/ifcfg-bond0 And select according to the usage mode The value is as follows. Since VLAN is to be reconfigured on bond0, IP and other relevant parameters are not configured. If IP and other parameters are configured, bond0 can be considered as a normal network card.


DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="miimon=100 mode=0"

Can be achieved by cat /proc/net/bonding/bond0 Depending on the configuration

Configure vlan on bond0, where the vlan number is 136, so the file name is /etc/sysconfig/network-scripts/ifcfg-bond0.136 , the content is as follows:


VLAN=yes
TYPE=Ethernet
DEVICE=bond0.136
NAME=bond0.136
PHYSDEV=bond0
ONBOOT=yes
BOOTPROTO=static
BRIDGE=docker0

Since docker bridge(docker0) will need to be attached to this vlan later, TYPE must be Ethernet with additional configuration BRIDGE=docker0

Configure bridge for docker ( /etc/sysconfig/network-scripts/ifcfg-docker0 ), the name can also be used, in order to reduce the docker engine need to increase the -b startup parameter, so the default name is used. IP is also not configured for docker0 because there is another ip of vlan to manage the host


TYPE=bridge
VLAN=yes
DEVICE=docker0
SLAVE=bond0.136
NAME=docker0
ONBOOT=yes
BOOTPROTO=none

Docker configuration

Modify the Docker startup file for machine use /usr/lib/systemd/system/docker.service , as follows:

After the line ExecStart=/usr/bin/docker daemon H fd:// add:


   --fixed-cidr=172.20.56.16/28 --default-gateway=172.20.56.1 --registry-mirror=http://registry.xxxx.com:5000 --insecure-registry=docker.xxx.com:5000 --storage-driver=overlay --ip-forward=false --iptables=false --log-driver=journald
    (note: "--fixed-cidr=" The machine container is filled in the back IP Subnet segment; "--default-gateway=" The container is in the back IP Subnet gateway; "--registry-mirror=" The information in the back is for the production environment docker registry The domain name.) 

Here are some pits:

1. For the storage of Docker, do not use the default mode, but use overlay and CentOS 7.X

2. Since the network configuration is not in the mode of Bond when the machine is installed, the original configuration needs to be cleaned after the configuration of bond is completed

Modify /etc/ sysctl.conf to net.ipv4.ip_forward The value of "is changed to 1

There is no need to configure gateway on the /etc/sysconfig/network file

Create a file/etc modules - load. d/bonding conf, content is as follows: bonding

3. By default, after installing docker, there will be no docker user group or docker user, so you must use root. If you don't use root, you need:

Create docker user group: sudo groupadd docker

Add the current user to the docker user group, for example, the current apps user: SLAVE=yes
0

Create docker users and join the docker user group: sudo useradd docker -g docker

Modify the "/var/lib/docker" directory and its subdirectories owner and group: sudo chown -R docker:docker

Log out and log in again to refresh the current user's permissions.

Pit is our biggest in some platform services such as Zookeeper installed on host 1, if the container want to access the configuration for the container on the host service is access, and the fixed for a long time have no way to find a network of colleagues, from the switch configuration and so on all have no problem, then under the guidance of Mr. Luo do try the following:

- start tcpdump on the host and grab the ping package from the container on the host. It can be found that the host has received the ICMP package from the container, but the container has not received any response. ping always times out.

a, open the Mars text test to see if any packages come in: sudo sysctl net.ipv4.conf.bond0/51.log_martians=1 (bond0/51 is the network card corresponding to bond0.51) you can see that there are packets coming in indicating that there are packets of unknown origin.

From the above two points, it can be thought that we have configured two different network CARDS on the network, different network breaks, in principle, they should be isolated, and the data to access network segment 1 should come in from the network card of network segment 1. If it comes in from network segment 2, Linux considers it as an illegal packet

d, according to the reason, is due to the problem of RP(Reverse Path) filtering of Linux, in this case, it is necessary to shut down RP of this host


sudo sysctl net.ipv4.conf.bond0/51.rp_filter=0
sudo sysctl net.ipv4.conf.bond0/52.rp_filter=0
sudo sysctl net.ipv4.conf.all.rp_filter=0
sudo sysctl net.ipv4.conf.bond0.rp_filter=0

Third party network Plugin

At present the better choice is the following several, but still groping

Calico, http://projectcalico.org/

Contiv, http://docs.contiv.io

conclusion

The above is the detailed introduction of VLAN network mode configuration in Docker. I hope all the contents of this article can be helpful for you to learn or use Docker. If you have any questions, please leave a message to communicate.


Related articles: