Realization of k8s node Node Rejoining master Cluster

1. Delete the node node

Execute kubectl delete node node01

2. If you join directly at this time, you will report an error. As follows:

[root@k8s-node02 pki]# kubeadm join 192.168.140.128:6443 --token abcdef.0123456789abcdef   --discovery-token-ca-cert-hash sha256:a3d9827be411208258aea7f3ee9aa396956c0a77c8b570503dd677aa3b6eb6d8 
[preflight] Running pre-flight checks
 [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.12. Latest validated version: 18.09
error execution phase preflight: [preflight] Some fatal errors occurred:
 [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
 [ERROR FileAvailable--etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf already exists
 [ERROR Port-10250]: Port 10250 is in use
 [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

Solution:

According to the error report, you can see that the port is occupied, and the configuration file can be ca certificate has been generated, so you need to delete these configuration files and certificates, and kill drops the occupied port. It is recommended to back up before deleting.

[root@k8s-node02 pki]# lsof -i:10250
COMMAND PID USER  FD  TYPE DEVICE SIZE/OFF NODE NAME
kubelet 694 root  30u IPv6 26021   0t0 TCP *:10250 (LISTEN)
[root@k8s-node02 pki]# kill -9 694
[root@k8s-node02 pki]# cd /etc/kubernetes/
[root@k8s-node02 kubernetes]# ls
bootstrap-kubelet.conf kubelet.conf manifests pki
[root@k8s-node02 kubernetes]# mv bootstrap-kubelet.conf bootstrap-kubelet.conf_bk
[root@k8s-node02 kubernetes]# mv kubelet.conf kubelet.conf_bk
[root@k8s-node02 kubernetes]# cd pki/
[root@k8s-node02 pki]# ls
ca.crt
[root@k8s-node02 pki]# rm -rf ca.crt 

3. Join again, and an error is reported again.

[root@k8s-node02 ~]# kubeadm join 192.168.140.128:6443 --token abcdef.0123456789abcdef   --discovery-token-ca-cert-hash sha256:a3d9827be411208258aea7f3ee9aa396956c0a77c8b570503dd677aa3b6eb6d8 
[preflight] Running pre-flight checks
 [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.12. Latest validated version: 18.09
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
error execution phase kubelet-start: error uploading crisocket: timed out waiting for the condition

Solution:

Perform kubeadm reset child node reset

[root@k8s-node02 ~]# kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0710 10:22:57.487306  31093 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

4. Finally, join and solve the problem.

[root@k8s-node02 ~]# kubeadm join 192.168.140.128:6443 --token abcdef.0123456789abcdef   --discovery-token-ca-cert-hash sha256:a3d9827be411208258aea7f3ee9aa396956c0a77c8b570503dd677aa3b6eb6d8 
[preflight] Running pre-flight checks
 [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.12. Latest validated version: 18.09
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

5. Check the master node and join successfully.

[root@k8s-master01 ~]# kubectl get nodes
NAME      STATUS  ROLES  AGE  VERSION
k8s-master01  Ready  master  120m  v1.15.1
k8s-node01   Ready  <none>  100m  v1.15.1
k8s-node02   Ready  <none>  83m  v1.15.1