Solution of nginx ingress controller log persistence solution
- 2020-05-24 06:48:37
- OfStack
Recently, I saw an article about the application of nginx-ingress-controller on a public account. The following person comments on how to do log persistence, just met the problem in the work, tidy up a scheme, just for reference.
nginx ingress - controller log
The log of nginx-ingress-controller consists of three parts:
controller log: output to stdout, which can be configured to output to files via the start parameter, through the wok of log_dir. Redirection to files will be automatically rotated, but not automatically cleaned up accesslog: output to stdout, which file can be configured through the fields in nginx-configuration. Output to a file does not automatically rotate or clean up errorlog: output to stderr, configured similarly to accesslog.
Drop disk to controller log
The log of controller needs to be cleaned regularly. Because controller log is through klog (k8s io/klog) of the output, to log rolling, so we regularly cleaned 1 through script before time log file.
Drop to nginx log
Modify configmap: nginx-configuration. Configure the output paths for accesslog and errorlog, replacing the default stdout and stderr. Output path we can send with controller1, easy to find.
Both accesslog and errorlog have only one log file, so we can use logrotate for logging rotation, rotating and cleaning the logs that are output to the host. Configuration such as:
$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
su root list
rotate 7
daily
maxsize 50M
copytruncate
missingok
create 0644 www-data root
}
In the official templates, nginx-ingress-controller is logged in to the launch container as 33 by default, so there is a permission issue when mounting the hostpath path. We need to manually execute chown-R 33:33 /data/log/nginx_ingress_controller on the machine.
Automation ops
nginx log plate, the 2nd and 3rd both need manual operation and maintenance, is there any solution?
The crux of the matter is: is there any way to add an hook before the nginx-ingress-controller container starts to execute chown in the host's designated directory?
You can use initContainer. initcontainer must run and exit successfully before the container in containers runs. Using the 1k8s feature, we developed an docker image, which only executes the following script:
#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown -R $userID:$userID $logdir
The script reads 1 of the environment variables and determines which directory needs to be changed to user group.
Package the script as dockerimage and place it in deploy yaml of nginx-ingress-controller as initcontainers. Note that the environment variables and volumeMount are configured for this initcontainer.
On the second point, we notice that nginx-ingress-controller has logrotate in its base image, so the problem is simple. We simply mount the written logrotate configuration file into the container as configmap.
1 deploy yaml is as follows:
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: kube-system
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: default
namespace: kube-system
spec:
backend:
serviceName: default-http-backend
servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: kube-system
labels:
app: ingress-nginx
data:
use-forwarded-headers: "true"
# This configuration nginx The redirection target of the log
access-log-path: /var/log/nginx_ingress_controller/access.log
error-log-path: /var/log/nginx_ingress_controller/error.log
---
# create 1 a configmap , the configuration nginx Log rotation strategy, corresponding to nginx Log the log file in the container
apiVersion: v1
data:
nginx.log: |
{{ user_nginx_log.host_path }}/access.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
{{ user_nginx_log.host_path }}/error.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
kind: ConfigMap
metadata:
name: nginx-ingress-logrotate
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ingress-nginx
namespace: kube-system
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
tolerations:
- key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "system/ingress"
operator: In
values:
- "true"
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
# configuration initcontainer To ensure the nginx-ingress-controller Configure the permissions for the log directory before the container starts
initContainers:
- name: adddirperm
image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
env:
- name: LOG_DIR
value: /var/log/nginx_ingress_controller
- name: USER_ID
value: "33"
volumeMounts:
- name: logdir
mountPath: /var/log/nginx_ingress_controller
containers:
- name: nginx-ingress-controller
image: "{{ image_registry.addr }}/{{ image.ingress }}"
imagePullPolicy: IfNotPresent
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
# Set up the controller The output path and mode of the log
- --log_dir=/var/log/nginx_ingress_controller
- --logtostderr=false
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
resources:
requests:
cpu: 100m
memory: 256Mi
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
# Configure the mount container for the controller component and nginx Log output path
- name: logdir
mountPath: /var/log/nginx_ingress_controller
# configuration nginx The log logrotate Configure the mount path
- name: logrotateconf
mountPath: /etc/logrotate.d/nginx.log
subPath: nginx.log
volumes:
# Controller components and nginx The log output path is that of the host machine hostpath
- name: logdir
hostPath:
path: {{ user_nginx_log.host_path }}
type: ""
# nginx The log rotation profile comes from configmap
- name: logrotateconf
configMap:
name: nginx-ingress-logrotate
items:
- key: nginx.log
path: nginx.log
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
tolerations:
- key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "system/ingress"
operator: In
values:
- "true"
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: "{{ image_registry.addr }}/{{ image.http_backend }}"
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
Finally, some people suggest that initcontainer be removed and replaced with a layer of layer based on the original nginx-ingress-controller image, where the script to configure path permissions is executed. Personally, I think this method is neither beautiful nor convenient. The only benefit of 1 is that deploy yaml is still simple (but not as simple as volumeMount). But still see individual use feeling ~