k8s 部署node应用，ingress-nginx映射公网访问（centos8.x最新验证）

k8s部署node express web应用本文档用于梳理k8s部署node应用的过程准备项目项目名称开放端口路由1路由2websvr13000/web1/index/web1/sendwebsvr23001/web2/index/web2/send为了快速搭建，此处的websvr采用express的脚手架express-generator安装：#安装express-generator：$ np

吃方便面的猫

3386人浏览 · 2021-10-20 10:11:37

吃方便面的猫 · 2021-10-20 10:11:37 发布

k8s部署node express web应用

本文档用于梳理k8s部署node应用的过程
关于k8s的部署步骤可以参考：《k8s部署，亲测有效，无坑》

准备项目

项目名称	开放端口	路由1	路由2
websvr1	3000	/web1/index	/web1/send
websvr2	3001	/web2/index	/web2/send

为了快速搭建，此处的websvr采用express的脚手架express-generator安装：

#安装express-generator：
$ npm install express-generator -g  
#创建脚手架应用app
$ express app
#安装依赖
$ cd app && npm install

在app同级目录编辑Dockerfile:

#指定node版本
FROM node:10.15.1	
#指定作者
MAINTAINER [SCH]
#将同级app文件夹添加进入docker下指定目录
ADD app /opt/app
#指定工作目录
WORKDIR /opt/app
#指定对外端口
EXPOSE 3000
#启动执行命令
CMD [ "nohup","npm","start", "&"]

将app与同级Dockerfile一起创建文件夹，命名为websvr1，修改app/routes/index.js路由文件：

//此处定义了两个get接口与一个post接口，
router.get('/web1/index', function (req, res, next) {
    res.render('index', {title: 'Express1'});
});

router.post('/web1/getIndex', function (req, res, next) {
    res.send("get index1");
});

//通过websvr1向websvr2，service发起请求，用来验证k8s，service之间的通信流程
router.get('/web1/send', function (req, res, next) {
    request({
        url: `http://websvr2-service:3001/web2/getIndex`,
        method: "POST",
        timeout: 10000
    }, (error, response, body) => {
        if (error) {
            console.log(error);
            res.render('index', {title: "请求失败1"});
            return
        }
        res.render('index', {title: body});
    })
});

另外拷贝一份websvr，将app目录下/bin/www内的默认端口和Dockerfile对外端口统一修改为3001，压缩命名为websvr2：

//此处定义了两个get接口与一个post接口，
router.get('/web2/index', function (req, res, next) {
    res.render('index', {title: 'Express2'});
});

router.post('/web2/getIndex', function (req, res, next) {
    res.send("get index2");
});

//通过websvr1向websvr2，service发起请求，用来验证k8s，service之间的通信流程
router.get('/web2/send', function (req, res, next) {
    request({
        url: `http://websvr1-service:3000/web1/getIndex`,
        method: "POST",
        timeout: 10000
    }, (error, response, body) => {
        if (error) {
            console.log(error);
            res.render('index', {title: "请求失败2"});
            return
        }
        res.render('index', {title: body});
    })
});

k8s集群条件

节点名称	IP
k8s-master	172.16.66.169
k8s-node1	172.16.66.168
k8s-node2	172.16.66.170

创建docker镜像

将websvr1，websvr2分别上传至node1,node2下的/opt目录下，并创建docker镜像

$ cd /opt/websvr1
$ docker build -t websvr:v1 .

$ cd /opt/websvr2
$ docker build -t websvr:v2 .

#查看docker镜像
$ docker images

REPOSITORY                                                        TAG        IMAGE ID       CREATED          SIZE
websvr                                                            v2         2a61bbea0d63   16 seconds ago   907MB
websvr                                                            v1         a3adb933da80   32 seconds ago   907MB
calico/node                                                       v3.20.1    355c1ee44040   4 weeks ago      156MB
calico/pod2daemon-flexvol                                         v3.20.1    55fa5eb71e09   4 weeks ago      21.7MB
calico/cni                                                        v3.20.1    e69ccb66d1b6   4 weeks ago      146MB
registry.aliyuncs.com/google_containers/kube-apiserver            v1.21.0    4d217480042e   6 months ago     126MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.21.0    38ddd85fe90e   6 months ago     122MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.21.0    62ad3129eca8   6 months ago     50.6MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.21.0    09708983cc37   6 months ago     120MB
registry.aliyuncs.com/google_containers/pause                     3.4.1      0f8457a4c2ec   9 months ago     683kB
coredns/coredns                                                   1.8.0      296a6d5035e2   12 months ago    42.5MB
registry.aliyuncs.com/google_containers/coredns/coredns           v1.8.0     296a6d5035e2   12 months ago    42.5MB
registry.aliyuncs.com/google_containers/etcd                      3.4.13-0   0369cf4303ff   13 months ago    253MB
node                                                              10.15.1    8fc2110c6978   2 years ago      897MB

k8s部署websvr

这里使用k8s，deployment，service部署websvr

deployment：创建docker容器群，对于同一个websvr，可以创建多个相同副本，通过分配不同虚拟IP及端口进行访问

service：当存在多个websvr容器副本后，如何通过统一的入口对多个websvr进行访问，就需要使用到service，可以简单理解为对多个容器副本的封装

$ vim websvr1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: websvr1-deployment
spec:
  selector:
    matchLabels:
      app: websvr1
  replicas: 3
  template:
    metadata:
      labels:
        app: websvr1
    spec:
      containers:
      - name: websvr1
        image: websvr:v1
        ports:
        - containerPort: 3000
       
---

apiVersion: v1
kind: Service
metadata:
  name: websvr1-service
spec:
  selector:
    app: websvr1
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

$ kubectl apply -f websvr1.yaml
$ kubectl get pods -o wide 

NAME                                  READY   STATUS    RESTARTS   AGE    IP               NODE        NOMINATED NODE   READINESS GATES
websvr1-deployment-7cb5776d76-mzx96   1/1     Running   0          3m8s   10.244.169.134   k8s-node2   <none>           <none>
websvr1-deployment-7cb5776d76-nzx7w   1/1     Running   0          3m8s   10.244.36.68     k8s-node1   <none>           <none>
websvr1-deployment-7cb5776d76-zzhdb   1/1     Running   0          3m8s   10.244.169.135   k8s-node2   <none>           <none>

同样方法部署websvr2，将暴露端口改为3001：

$ vim websvr2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: websvr2-deployment
spec:
  selector:
    matchLabels:
      app: websvr2
  replicas: 3
  template:
    metadata:
      labels:
        app: websvr2
    spec:
      containers:
      - name: websvr2
        image: websvr:v2
        ports:
        - containerPort: 3001
       
---

apiVersion: v1
kind: Service
metadata:
  name: websvr2-service
spec:
  selector:
    app: websvr2
  ports:
  - protocol: TCP
    port: 3001
    targetPort: 3001

$ kubectl apply -f websvr2.yaml
$ kubectl get pods -o wide 

NAME                                  READY   STATUS    RESTARTS   AGE     IP               NODE        NOMINATED NODE   READINESS GATES
websvr1-deployment-7cb5776d76-mzx96   1/1     Running   0          7m35s   10.244.169.134   k8s-node2   <none>           <none>
websvr1-deployment-7cb5776d76-nzx7w   1/1     Running   0          7m35s   10.244.36.68     k8s-node1   <none>           <none>
websvr1-deployment-7cb5776d76-zzhdb   1/1     Running   0          7m35s   10.244.169.135   k8s-node2   <none>           <none>
websvr2-deployment-58c8b7ffcd-57tsz   1/1     Running   0          7s      10.244.36.69     k8s-node1   <none>           <none>
websvr2-deployment-58c8b7ffcd-9lg4c   1/1     Running   0          7s      10.244.36.70     k8s-node1   <none>           <none>
websvr2-deployment-58c8b7ffcd-dgzl5   1/1     Running   0          7s      10.244.36.71     k8s-node1   <none>           <none>

验证

对于运行在各个node节点上的pod，通过统一的serviceIP及端口进行访问，service通过一定的负载均衡规则，分发到不同的node节点的pod上进行业务处理

$ kubectl get svc -o wide

NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     SELECTOR
kubernetes        ClusterIP   10.96.0.1        <none>        443/TCP    135m    <none>
websvr1-service   ClusterIP   10.102.171.58    <none>        3000/TCP   10m     app=websvr1
websvr2-service   ClusterIP   10.104.188.128   <none>        3001/TCP   2m34s   app=websvr2

#发现此处有刚刚创建的两个websvr service，分别对应websvr1:3000及websvr2:3001

此时外网还无法访问k8s集群内容器，接下来需要进一步部署ingress

ingress-nginx部署

Ingress-nginx version	k8s supported version	Alpine Version	Nginx Version
v0.48.1	1.21, 1.20, 1.19	3.13.5	1.20.1
v0.47.0	1.21, 1.20, 1.19	3.13.5	1.20.1
v0.46.0	1.21, 1.20, 1.19	3.13.2	1.19.6

在master及所有node执行：

# 从阿里云镜像仓库拉取ingress-nginx所需版本：
$ docker pull registry.cn-hangzhou.aliyuncs.com/kubernetes-fan/ingress-nginx:v0.48.1

# 将阿里云镜像重新打tag命名为官方镜像名：
$ docker tag registry.cn-hangzhou.aliyuncs.com/kubernetes-fan/ingress-nginx:v0.48.1 k8s.gcr.io/ingress-nginx/controller:v0.48.1

# 删除阿里云镜像：
$ docker rmi registry.cn-hangzhou.aliyuncs.com/kubernetes-fan/ingress-nginx:v0.48.1

打开ingress-nginx 0.48.1的deploy.yaml网站、将yaml内容全部复制到本地。

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/baremetal/deploy.yaml

修改本地的deploy.yaml文件：

image: k8s.gcr.io/ingress-nginx/controller:v0.48.1@sha256:e9fb216ace49dfa4a5983b183067e97496e7a8b307d2093f4278cd550c303899
# 修改为
image: k8s.gcr.io/ingress-nginx/controller:v0.48.1

外网无法打开可以使用下面保存的yaml文件：

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
#kind: Deployment
#apiVersion: extensions/v1beta1
# 修改为DaemonSet类型，随每个node节点创建和删除，配合污点容忍可以实现ingress-nginx高可用
kind: DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      #开启本机网络
      hostNetwork: true
      containers:
        - name: controller
          image: k8s.gcr.io/ingress-nginx/controller:v0.48.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
            #若本机端口被占用，需要另行设置
            #- --http-port=81
            #- --https-port=1444
            #- --status-port=18081
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1beta1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1beta1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-3.34.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.48.1
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: docker.io/jettech/kube-webhook-certgen:v1.5.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.34.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.48.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-3.34.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.48.1
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: docker.io/jettech/kube-webhook-certgen:v1.5.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

在master执行：

$ kubectl apply -f deploy.yaml

$ kubectl get pod -o wide -n ingress-nginx

NAME                                   READY   STATUS      RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-87rgx   0/1     Completed   0          72s   10.244.169.137   k8s-node2   <none>           <none>
ingress-nginx-admission-patch-hq6b6    0/1     Completed   0          72s   10.244.36.74     k8s-node1   <none>           <none>
ingress-nginx-controller-f7d7r         1/1     Running     0          72s   172.16.66.170    k8s-node2   <none>           <none>
ingress-nginx-controller-p2z5t         1/1     Running     0          72s   172.16.66.168    k8s-node1   <none>           <none>

#可以看到ingress已经跟随node节点创建了两个controller用以监听nginx配置文件变化并更新

配置ingress

ingress-nginx安装完成后，还需要配置ingress路由规则，类似nginx的路由规则：

$ vim ingressRule.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: k8s.test.com						#指定域名
      http:			
        paths:
        - path: /web1							#一级路由				
          pathType: Prefix						#匹配规则 Prefix：前缀
          backend:
            service:
              name: websvr1-service				#指向的service
              port: 
                number: 3000					#对应的service暴露的端口
        - path: /web2
          pathType: Prefix
          backend:
            service:
              name: websvr2-service
              port: 
                number: 3001

$ kubectl apply -f ingressRule.yaml

$ kubectl describe ingress

Name:             my-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  k8s.scbczx.com  
                  /web1   websvr1-service:3000 (10.244.169.134:3000,10.244.169.135:3000,10.244.36.68:3000)
                  /web2   websvr2-service:3001 (10.244.169.136:3001,10.244.36.72:3001,10.244.36.73:3001)
Annotations:      kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    11s   nginx-ingress-controller  Scheduled for sync
  Normal  Sync    11s   nginx-ingress-controller  Scheduled for sync

验证

此时通过curl发起get请求验证ingress-nginx路由规则

$ curl k8s.test.com/web1/index
<!DOCTYPE html><html><head><title>Express1</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>Express1</h1><p>Welcome to Express1</p></body></html>
$ curl k8s.test.com/web1/send
<!DOCTYPE html><html><head><title>get index2</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>get index2</h1><p>Welcome to get index2</p></body></html>

$ curl k8s.test.com/web2/index
<!DOCTYPE html><html><head><title>Express2</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>Express2</h1><p>Welcome to Express2</p></body></html>
$ curl k8s.test.com/web2/send
<!DOCTYPE html><html><head><title>get index1</title><link rel="stylesheet" href="/stylesheets/style.css"></head><body><h1>get index1</h1><p>Welcome to get index1</p></body></html>

至此，k8s集群内的websvr都成功的通过公网域名进行访问

附：在实际的项目进展中，存在一些在当前线程内存中存储用户登录态的情况，比如服务器session，如果按照当前的service分发规则，很有可能导致用户登录session丢失的问题，那么service是否可以像nginx一样配置分发规则，比如按照前端IP？后面会在别的文档内单独讨论。

如有问题，欢迎指正。

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub