1. Ingress介绍

Service对集群之外暴露服务的主要方式有两种：NodePort和LoadBalancer，但这两种方式都有缺点：

• NodePort方式会占用集群每个节点的很多端口
• LoadBalancer每个Service都需要一个LB，并且需要kubernetes之外的设备支持

Ingress只需要一个NodePort或者一个LB就可以满足暴露多个Service的需求，工作机制大致如下图所示

Ingress相当于一个七层的负载均衡器，是kubernetes对反向代理的一个抽象，它的工作原理类似于Nginx

原理是在Ingress Service中定义哪个域名对应kubernetes集群中的哪个Service，Ingress Controller通过监听这些映射规则并转化为Nginx(或Contour、Haproxy)的反向代理配置，并将反向代理配置写入这些服务，最后由Nginx(或Contour、Haproxy)对外提供服务

2. ingress-nginx Controller安装

更多细节参考github kubernetes/ingress-nginx

这里安装的是ingress-nginx 1.2.0版本，适用于Kubernetes 1.23，使用的Nginx版本1.19.10

1. 下载Deployent的yaml文件

[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/cloud/deploy.yaml

2. 在所有node节点上，拉取国内的ingress-nginx/controller:v1.2.0镜像源

[root@k8s-node1 ~]# crictl pull dyrnq/controller:v1.2.0

3. 修改yaml文件内容，将下面两处

image: k8s.gcr.io/ingress-nginx/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185

image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

修改成

image: docker.io/dyrnq/controller:v1.2.0@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185

image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1@sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660

4. 运行deploy

[root@k8s-master ~]# kubectl apply -f deploy.yaml 
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
[root@k8s-master ~]# 

5. 查看ingress-nginx的pod

[root@k8s-master ~]# kubectl get pod -n ingress-nginx -o wide
NAME                                        READY   STATUS      RESTARTS       AGE   IP               NODE        NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-crt25        0/1     Completed   0              37h   10.244.169.172   k8s-node2   <none>           <none>
ingress-nginx-admission-patch-9nn5g         0/1     Completed   1              37h   10.244.169.170   k8s-node2   <none>           <none>
ingress-nginx-controller-77d4dc6978-cksgf   1/1     Running     1 (151m ago)   37h   10.244.169.175   k8s-node2   <none>           <none>
[root@k8s-master ~]# 

6. 查看ingress-nginx的Service

[root@k8s-master ~]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.68.173   <pending>     80:31238/TCP,443:30986/TCP   37h
ingress-nginx-controller-admission   ClusterIP      10.96.185.8    <none>        443/TCP                      37h
[root@k8s-master ~]#

3. 准备Pod和Service

3.1 准备Pod

通过Deployment分别创建3个nginx pod和3个tomcat pod。新建nginx-tomcat-deployment.yaml文件，内容如下。然后创建deployment

[root@k8s-master ~]# cat nginx-tomcat-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-pod
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
        - name: nginx-container
          image: nginx:latest
          ports:
            - name: nginx-port
              containerPort: 80
              protocol: TCP

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  namespace: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat-pod
  template:
    metadata:
      labels:
        app: tomcat-pod
    spec:
      containers:
        - name: tomcat-container
          image: tomcat:8.5-jre10-slim
          ports:
            - name: tomcat-port
              containerPort: 8080
              protocol: TCP
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f nginx-tomcat-deployment.yaml 
deployment.apps/nginx-deployment created
deployment.apps/tomcat-deployment created
[root@k8s-master ~]#

查看Pod

[root@k8s-master ~]# kubectl get pod -n dev
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-84d58db8c4-75z5v   1/1     Running   0          24m
nginx-deployment-84d58db8c4-lqpqj   1/1     Running   0          24m
nginx-deployment-84d58db8c4-skcnq   1/1     Running   0          24m
tomcat-deployment-d4b4bdfdf-5jlqv   1/1     Running   0          24m
tomcat-deployment-d4b4bdfdf-lrbpt   1/1     Running   0          24m
tomcat-deployment-d4b4bdfdf-x7c97   1/1     Running   0          24m
[root@k8s-master ~]# 

3.2 准备Service

这里创建HeadLiness类型的Service分别指向nginx-pod和tomcat-pod，直接从Service的域名转发到后端Pod，不经过VIP(虚拟IP)。新建nginx-tomcat-headliness.yaml文件，内容如下。然后创建Service

[root@k8s-master ~]# cat nginx-tomcat-headliness.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: dev
spec:
  selector:
    app: nginx-pod
  type: ClusterIP
  clusterIP: None
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

---


apiVersion: v1
kind: Service
metadata:
  name: tomcat-service
  namespace: dev
spec:
  selector:
    app: tomcat-pod
  type: ClusterIP
  clusterIP: None
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080


[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f nginx-tomcat-headliness.yaml 
service/nginx-service created
service/tomcat-service created
[root@k8s-master ~]# 

查看Service

[root@k8s-master ~]# kubectl get svc -n dev
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
nginx-service    ClusterIP   None         <none>        80/TCP     4m1s
tomcat-service   ClusterIP   None         <none>        8080/TCP   4m1s
[root@k8s-master ~]#

4. Ingress的HTTP代理

4.1 Ingress的创建

新建ingress-http.yaml文件，内容如下。然后运行ingress

[root@k8s-master ~]# cat ingress-http.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-http
  namespace: dev
spec:
  rules:
    - host: nginx.bulut.com
      http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: nginx-service
                port:
                  number: 80  
    - host: tomcat.bulut.com
      http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: tomcat-service
                port:
                  number: 8080
  ingressClassName: nginx
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl apply -f ingress-https.yaml 
ingress.networking.k8s.io/ingress-http created
[root@k8s-master ~]#

4.2 Ingress的查看

[root@k8s-master ~]# kubectl get ingress ingress-http -n dev
NAME           CLASS   HOSTS                              ADDRESS   PORTS   AGE
ingress-http   nginx   nginx.bulut.com,tomcat.bulut.com             80      64m
[root@k8s-master ~]# 

4.3 Ingress详情的查看

[root@k8s-master ~]# kubectl describe ingress ingress-http -n dev
Name:             ingress-http
Labels:           <none>
Namespace:        dev
Address:          
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host              Path  Backends
  ----              ----  --------
  nginx.bulut.com   
                    /   nginx-service:80 (10.244.169.178:80,10.244.169.179:80,10.244.36.70:80)
  tomcat.bulut.com  
                    /   tomcat-service:8080 (10.244.169.176:8080,10.244.169.177:8080,10.244.36.69:8080)
Annotations:        <none>
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    116s (x3 over 4m34s)  nginx-ingress-controller  Scheduled for sync
[root@k8s-master ~]# 

4.4 nginx和tomcat的服务访问

在要访问的服务器的/etc/hosts文件中，添加IP域名映射。其中IP是Kubernetes运行ingress-nginx-controller pod的节点(这里是192.168.23.162/k8s-node2)，如下所示

[root@docker ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

......省略部分......

192.168.23.162 nginx.bulut.com
192.168.23.162 tomcat.bulut.com

[root@docker ~]#

然后访问nginx和tomcat

[root@docker ~]# 
[root@docker ~]# curl http://nginx.bulut.com:31238
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@docker ~]# 
[root@docker ~]# curl http://tomcat.bulut.com:31238




<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.35</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>
        <div id="wrapper">
            <div id="navigation" class="curved container">
                <span id="nav-home"><a href="https://tomcat.apache.org/">Home</a></span>
                <span id="nav-hosts"><a href="/docs/">Documentation</a></span>
                <span id="nav-config"><a href="/docs/config/">Configuration</a></span>
                <span id="nav-examples"><a href="/examples/">Examples</a></span>
                <span id="nav-wiki"><a href="https://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>
                <span id="nav-lists"><a href="https://tomcat.apache.org/lists.html">Mailing Lists</a></span>
                <span id="nav-help"><a href="https://tomcat.apache.org/findhelp.html">Find Help</a></span>
                <br class="separator" />
            </div>
            <div id="asf-box">
                <h1>Apache Tomcat/8.5.35</h1>
            </div>
......省略部分......
        </div>
    </body>

</html>
[root@docker ~]# 

4.5 Client到Pod的服务访问流程说明

这里以访问nginx服务为例进行说明：

1. Client访问http://nginx.bulut.com:31238，被解析成访问http://192.168.23.162:31238
2. 查看ingress-nginx的Service如下

[root@k8s-master ~]# kubectl get svc -n ingress-nginx 
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.68.173   <pending>     80:31238/TCP,443:30986/TCP   38h
ingress-nginx-controller-admission   ClusterIP      10.96.185.8    <none>        443/TCP                      38h
[root@k8s-master ~]#

虽然TYPE为LoadBalancer，但请求还是会转发到http://10.96.68.173:80

3. 因为我们最先访问的域名是nginx.bulut.com，所以会将此请求转发到http://nginx-service:80
4. 然后由3个nginx-pod中的一个提供服务

5. Ingress的HTTPS代理

5.1 生成证书

[root@k8s-master ~]# openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 3650 -out tls.crt -subj "/C=CN/ST=HN/L=CS/O=bulut/OU=bulut/CN=bulut.com/emailAddress=bulut@163.com"
Generating a 2048 bit RSA private key
............................+++
......................................................................................................+++
writing new private key to 'tls.key'
-----
[root@k8s-master ~]# 

参数说明如下：

• req：表示证书请求的子命令
• -newkey rsa:2048：表示生成私钥
• -nodes：表示私钥不加密
• -keyout tls.key：表示生成的私钥输出文件
• -x509：表示输出证书
• -days 3650：表示证书有效期
• -out tls.crt：表示生成的证书输出文件
• -subj “/C=CN/ST=HN/L=CS/O=bulut/OU=bulut/CN=bulut.com/emailAddress=bulut@163.com”：表示自动输入证书拥有者信息，分别表示：国家/省份/城市/公司名/部门名/域名/邮件地址

5.2 创建密钥

[root@k8s-master ~]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created
[root@k8s-master ~]#

5.1 Ingress的创建

新建ingress-https.yaml文件，内容如下。然后运行ingress

[root@k8s-master ~]# cat ingress-https.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-https
  namespace: dev
spec:
  tls:
    - secretName: tls-secret
      hosts:
        - nginx.bulut.com
        - tomcat.bulut.com
  rules:
    - host: nginx.bulut.com
      http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: nginx-service
                port:
                  number: 80
    - host: tomcat.bulut.com
      http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: tomcat-service
                port:
                  number: 8080

[root@k8s-master ~]#
[root@k8s-master ~]# kubectl apply -f ingress-https.yaml 
ingress.networking.k8s.io/ingress-https created
[root@k8s-master ~]#

4.2 Ingress的查看

[root@k8s-master ~]# kubectl get ingress ingress-https -n dev
NAME            CLASS    HOSTS                              ADDRESS   PORTS     AGE
ingress-https   <none>   nginx.bulut.com,tomcat.bulut.com             80, 443   2m38s
[root@k8s-master ~]#

4.3 Ingress详情的查看

[root@k8s-master ~]# kubectl get ingress ingress-https -n dev
NAME            CLASS    HOSTS                              ADDRESS   PORTS     AGE
ingress-https   <none>   nginx.bulut.com,tomcat.bulut.com             80, 443   2m38s
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl describe ingress ingress-https -n dev
Name:             ingress-https
Labels:           <none>
Namespace:        dev
Address:          
Ingress Class:    <none>
Default backend:  <default>
TLS:
  tls-secret terminates nginx.bulut.com,tomcat.bulut.com
Rules:
  Host              Path  Backends
  ----              ----  --------
  nginx.bulut.com   
                    /   nginx-service:80 (10.244.169.178:80,10.244.169.179:80,10.244.36.70:80)
  tomcat.bulut.com  
                    /   tomcat-service:8080 (10.244.169.176:8080,10.244.169.177:8080,10.244.36.69:8080)
Annotations:        <none>
Events:             <none>
[root@k8s-master ~]# 

4.4 nginx和tomcat的服务访问

在要访问的服务器的/etc/hosts文件中，添加IP域名映射。其中IP是Kubernetes运行ingress-nginx-controller pod的节点(这里是192.168.23.162/k8s-node2)，如下所示

[root@docker ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

......省略部分......

192.168.23.162 nginx.bulut.com
192.168.23.162 tomcat.bulut.com

[root@docker ~]#

然后访问nginx和tomcat

[root@docker ~]# 
[root@docker ~]# curl --insecure https://nginx.bulut.com:30986
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@docker ~]# 
[root@docker ~]# curl --insecure https://tomcat.bulut.com:30986



<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.35</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>
        <div id="wrapper">
            <div id="navigation" class="curved container">
                <span id="nav-home"><a href="https://tomcat.apache.org/">Home</a></span>
                <span id="nav-hosts"><a href="/docs/">Documentation</a></span>
                <span id="nav-config"><a href="/docs/config/">Configuration</a></span>
                <span id="nav-examples"><a href="/examples/">Examples</a></span>
                <span id="nav-wiki"><a href="https://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>
                <span id="nav-lists"><a href="https://tomcat.apache.org/lists.html">Mailing Lists</a></span>
                <span id="nav-help"><a href="https://tomcat.apache.org/findhelp.html">Find Help</a></span>
                <br class="separator" />
            </div>
            <div id="asf-box">
                <h1>Apache Tomcat/8.5.35</h1>
            </div>
......省略部分......
        </div>
    </body>

</html>
[root@docker ~]#