转载 https://blog.csdn.net/weixin_39961559/article/details/87935873

线上的k8s使用nginx所在的边缘节点来将外部访问导流到集群内部容器，ingress-nginx-controller是k8s众多ingress controller实现中的一种，以agent+nginx的方式提供服务。agent通过watch k8s的ingress、configmap、endpoint等资源的变化，修改nginx的配置文件，并负责nginx的reload等工作。如果ingress-nginx-controller插件中的nginx因某种原因需要升级，我们应该怎么办呢？(如nginx漏洞)
漏洞
遇到这个问题的时候，我第一个想到的是通过ingress-nginx-controller官方版本升级，于是去查询ingress-nginx-controller的github仓库查看插件版本对应的nginx版本，下图对应的是0.20.0的插件版本
22
虽然在官方的github仓库的最高版本是0.22.0，但是nginx版本是多少通过github并不清楚，而我又不希望通过下载每个官方镜像去查看nginx版本，并且这个nginx版本取决于官方的更新速度，这给个人的升级带来一定的不便，所以打算自己制作ingress-nginx-controller镜像。那思路是怎样的呢？既然我想替换nginx版本，那么我可以用当前在线上跑的插件版本，提取其中的ingress-nginx-controller组件，再根据我们想要升级的nginx版本就可以制作一个新的镜像了。想法虽然简单，但是，安装nginx指定版本，我们编译的时候要添加什么参数呢？这个我们可以通过当前版本的nginx -V去查看相关输出，我们能看到nginx的版本和编译相关信息。

nginx version: nginx/1.13.4
built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
built with OpenSSL 1.0.2g 1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock
–pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module
–with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module
–with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-file-aio --without-mail_pop3_module
–without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_module --without-http_scgi_module --with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic’
–add-module=/tmp/build/ngx_devel_kit-0.3.0 --add-module=/tmp/build/set-misc-nginx-module-0.31 --add-module=/tmp/build/nginx-module-vts-0.1.15 --add-module=/tmp/build/headers-more-nginx-module-0.32
–add-module=/tmp/build/nginx-goodies-nginx-sticky-module-ng-08a395c66e42 --add-module=/tmp/build/nginx-http-auth-digest-7955af9c77598c697ac292811914ce1e2b3b824c
–add-module=/tmp/build/ngx_http_substitutions_filter_module-bc58cb11844bc42735bbaef7085ea86ace46d05b

1
2
3
4
5
6
7
8
9
10
11
12
13
14
通过nginx -V我们能够清晰的看到编译nginx时我们的参数配置，以及所加载的模块，所以将nginx升级到1.14.2时也是通过这些参数去编译。
那么升级的思路主要是安装nginx了，为了做到nginx版本间无干扰，准备用直接从debian镜像开始构建

1、准备好nginx的源码包
wget http://nginx.org/download/nginx-1.14.2.tar.gz

2、准备nginx add module相关模块文件
相关文件
通过https://www.nginx.com/resources/wiki/modules/index.html查找编译选项中的add-module模块，将指定版本的nginx模块的github仓库下载到Dockerfile通目录下。

3、编译安装nginx
./configure --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log \
–error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid \
–http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
–http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi \
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module \
–with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module \
–with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module \
–with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module \
–with-threads --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module \
–without-http_uwsgi_module --without-http_scgi_module --with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic’
–add-module=/tmp/build/ngx_devel_kit-0.3.0 --add-module=/tmp/build/set-misc-nginx-module-0.31 --add-module=/tmp/build/nginx-module-vts-0.1.15
–add-module=/tmp/build/headers-more-nginx-module-0.32 --add-module=/tmp/build/nginx-goodies-nginx-sticky-module-ng-08a395c66e42
–add-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-module=/tmp/build/ngx_http_substitutions_filter_module-0.6.4

1
2
3
4
5
6
7
8
9
10
11
12
13
14
在编译之前要先确认好是否缺少某些依赖库，在我的实践过程中，我首先安装以下依赖

apt-get install openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev -y
1
也可以通过./configure检查缺少哪些依赖，并通过提示安装依赖。
编译通过之后就是

make && make install
cp /usr/share/nginx/sbin/nginx /usr/sbin/
1
2
所以整个过程将它整理成一个Dockerfile文件如下所示：

FROM debian:latest
MAINTAINER polarwu

RUN apt-get update && apt-get install wget openssl libssl-dev libpcre3 libpcre3-dev zlib1g-dev libgeoip-dev gcc make -y
WORKDIR /
RUN wget http://nginx.org/download/nginx-1.14.2.tar.gz && tar -zxvf nginx-1.14.2.tar.gz
RUN mkdir /var/log/nginx && chmod 755 /var/log/nginx && mkdir /var/lib/nginx && chmod 755 /var/lib/nginx
ADD ./ngx_devel_kit-0.3.0 /tmp/build/ngx_devel_kit-0.3.0
ADD ./set-misc-nginx-module-0.31 /tmp/build/set-misc-nginx-module-0.31
ADD ./nginx-module-vts-0.1.15 /tmp/build/nginx-module-vts-0.1.15
ADD ./headers-more-nginx-module-0.32 /tmp/build/headers-more-nginx-module-0.32
ADD ./nginx-goodies-nginx-sticky-module-ng-08a395c66e42 /tmp/build/nginx-goodies-nginx-sticky-module-ng-08a395c66e42
ADD ./nginx-http-auth-digest-1.0.0 /tmp/build/nginx-http-auth-digest-1.0.0
ADD ./ngx_http_substitutions_filter_module-0.6.4 /tmp/build/ngx_http_substitutions_filter_module-0.6.4
RUN cd /nginx-1.14.2 && ./configure --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log \
–error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid \
–http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
–http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi \
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module \
–with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module \
–with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module \
–with-http_sub_module --with-http_v2_module --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module \
–with-threads --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module \
–without-http_uwsgi_module --without-http_scgi_module --with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic’
–add-module=/tmp/build/ngx_devel_kit-0.3.0 --add-module=/tmp/build/set-misc-nginx-module-0.31 --add-module=/tmp/build/nginx-module-vts-0.1.15
–add-module=/tmp/build/headers-more-nginx-module-0.32 --add-module=/tmp/build/nginx-goodies-nginx-sticky-module-ng-08a395c66e42
–add-module=/tmp/build/nginx-http-auth-digest-1.0.0 --add-module=/tmp/build/ngx_http_substitutions_filter_module-0.6.4

RUN cd /nginx-1.14.2 && make && make install
RUN cp /usr/share/nginx/sbin/nginx /usr/sbin/
RUN mkdir /etc/nginx/template
RUN rm -rf /nginx-1.14.2*
ADD ./nginx.tmpl /etc/nginx/template/
ADD ./ingress-controller /ingress-controller
ADD ./nginx-ingress-controller /
ADD ./GeoIP.dat /etc/nginx
ADD ./GeoLiteCity.dat /etc/nginx
RUN rm -rf /tmp/build
RUN apt-get remove wget gcc make -y && apt-get clean
EXPOSE 80
EXPOSE 443
CMD ["/nginx-ingress-controller"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
最后编写一个yaml文件测试我们制作的镜像是否正常运行

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ingress-lb
labels:
name: nginx-ingress-lb
namespace: ingress-nginx
spec:
selector:
matchLabels:
name: nginx-ingress-lb
template:
metadata:
labels:
name: nginx-ingress-lb
spec:
hostNetwork: true
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
nginx-ingress-lb: ‘true’
containers:
- name: nginx-ingress-lb
image: nginx-ingress-controller:nginx-v1.14.2
readinessProbe:
httpGet:
path: /healthz
port: 80
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 80
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80
name: http
- containerPort: 443
hostPort: 443
name: https
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_MASTER
value: http://192.168.1.123:8080
args:
- /nginx-ingress-controller
- --default-backend-service=ingress-nginx/http-backend
- --tcp-services-configmap=default/nginx-tcp-ingress-configmap

apiVersion: v1
kind: Service
metadata:
name: nginx-ingress-lb
namespace: ingress-nginx
labels:
app: nginx-ingress-lb
spec:
ports:

• port: 80
  targetPort: 80
  protocol: TCP
  name: http
• port: 443
  targetPort: 443
  protocol: TCP
  name: https
  selector:
  name: nginx-ingress-lb

---

apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-tcp-ingress-configmap
namespace: default
data:
“8846”: default/helloworld:8777

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
通过访问10.142.233.79:8846(10.142.233.79为nginx所在节点)，我们升级的ingress-nginx-controller已经正常工作
hello

总结
文章介绍了通过升级nginx灵活配置ingress-nginx-controller插件下的nginx版本，并通过测试验证了我们nginx和ingress-controller无缝升级，解决了我们的nginx漏洞整改需求。一般情况下，ingress-controller实现的功能是基于nginx，所以我们在升级nginx版本时应首先通过nginx -V查看nginx的相关编译参数以便新版本按照现有nginx版本进行编译，提供现有ingress-nginx-controller插件的功能需求。