nginx-ingress-controller日志持久化方案的解决
这篇文章主要介绍了nginx-ingress-controller日志持久化方案的解决,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友们下面随着小编来一起学习学习吧
最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化,刚好工作上遇到该问题,整理一个方案,仅供参考。
nginx-ingress-controller的日志
nginx-ingress-controller的日志包括三个部分:
controller日志: 输出到stdout,通过启动参数中的–log_dir可已配置输出到文件,重定向到文件后会自动轮转,但不会自动清理
accesslog:输出到stdout,通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
errorlog:输出到stderr,配置方式与accesslog类似。
给controller日志落盘
给nginx-ingress-controller挂一个hostpath: /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ,
给nginx-ingress-controller配置log-dir和logtostderr参数,将日志重定向到/var/log/nginx_ingress_controller/中。
controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的,会进行日志滚动,所以我们通过脚本定时清理一定时间之前的日志文件即可。
给nginx日志落盘
修改configmap: nginx-configuration。配置accesslog和errorlog的输出路径,替换默认的stdout和stderr。输出路径我们可以与controller一致,便于查找。
accesslog和errorlog都只有一个日志文件,我们可以使用logrotate进行日志轮转,将输出到宿主机上的日志进行轮转和清理。配置如:
1 2 3 4 5 6 7 8 9 10 | $ cat /etc/logrotate.d/nginx.log /data/log/nginx_ingress_controller/access.log { su root list rotate 7 daily maxsize 50M copytruncate missingok create 0644 www-data root } |
官方提供的模板中,nginx-ingress-controller默认都是以33这个用户登录启动容器的,因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.
自动化ops
nginx日志落盘中,第2、3两点均需要人工运维,有什么解决办法吗?
问题的关键是:有什么办法可以在nginx-ingress-controller容器启动之前加一个hook,将宿主机的指定目录执行chown呢?
可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性,我们开发一个docker image,里面只执行如下脚本:
1 2 3 4 5 | #!/bin/bash logdir=$LOG_DIR userID=$USER_ID echo "try to set dir: $logdir 's group as $userID" chown -R $userID:$userID $logdir |
脚本读取一些环境变量, 确认需要修改哪个目录,改成怎样的user group。
将脚本打包成dockerimage, 放在nginx-ingress-controller的deploy yaml中,作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.
再说第二点,我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate,那么问题就简单了,我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。
一个deploy yaml如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 | --- apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: kube-system spec: type: ClusterIP ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app: ingress-nginx --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: kube-system labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: default namespace: kube-system spec: backend: serviceName: default-http-backend servicePort: 80 --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: kube-system labels: app: ingress-nginx data: use-forwarded-headers: "true" # 此处配置nginx日志的重定向目标 access-log-path: /var/log/nginx_ingress_controller/access.log error-log-path: /var/log/nginx_ingress_controller/error.log --- # 创建一个configmap,配置nginx日志的轮转策略,对应的是nginx日志在容器内的日志文件 apiVersion: v1 data: nginx.log: | {{ user_nginx_log.host_path }}/access.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root } {{ user_nginx_log.host_path }}/error.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root } kind: ConfigMap metadata: name: nginx-ingress-logrotate namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: kube-system rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: name: ingress-nginx namespace: kube-system spec: selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" dnsPolicy: ClusterFirstWithHostNet hostNetwork: true # 配置initcontainer,确保在nginx-ingress-controller容器启动前将日志目录的权限配置好 initContainers: - name: adddirperm image: "{{ image_registry.addr }}/{{ image.adddirperm }}" env: - name: LOG_DIR value: /var/log/nginx_ingress_controller - name: USER_ID value: "33" volumeMounts: - name: logdir mountPath: /var/log/nginx_ingress_controller containers: - name: nginx-ingress-controller image: "{{ image_registry.addr }}/{{ image.ingress }}" imagePullPolicy: IfNotPresent args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io # 设置controller日志的输出路径和方式 - --log_dir=/var/log/nginx_ingress_controller - --logtostderr=false securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 resources: requests: cpu: 100m memory: 256Mi livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: # 配置挂载容器中控制器组件和nginx的日志输出路径 - name: logdir mountPath: /var/log/nginx_ingress_controller # 配置nginx日志的logrotate配置挂载路径 - name: logrotateconf mountPath: /etc/logrotate.d/nginx.log subPath: nginx.log volumes: # 控制器组件和nginx的日志输出路径为宿主机的hostpath - name: logdir hostPath: path: {{ user_nginx_log.host_path }} type: "" # nginx日志的轮转配置文件来自于configmap - name: logrotateconf configMap: name: nginx-ingress-logrotate items: - key: nginx.log path: nginx.log --- apiVersion: apps/v1 kind: DaemonSet metadata: name: default-http-backend namespace: kube-system labels: app: default-http-backend spec: selector: matchLabels: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" containers: - name: default-http-backend # Any image is permissible as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: "{{ image_registry.addr }}/{{ image.http_backend }}" imagePullPolicy: IfNotPresent livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- |
最后,有的人建议将initcontainer去掉,改为基于原有的nginx-ingress-controller镜像加一层layer,将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观,也不方便。唯一的好处仅在于deploy yaml仍然简洁(但少不了volumeMount之类的配置)。不过还是看个人使用感受吧~
到此这篇关于nginx-ingress-controller日志持久化方案的解决的文章就介绍到这了