环境

基于记录 - k8s 入门搭建 (1.16.0, helloweb) 搭建的环境：
k8s集群master=k8s0=192.168.199.200，worker=k8s1=192.168.199.201，加worker=k8s2=192.168.199.202。

了解service type

参考 https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies

每个节点（包括master）的kube-proxy 负责实现service 的vip，有三种代理模式：
1）user space 模式
在当前节点（随机）设置一个端口，访问该端口即转发到service对应的某个pod。
另外设置iptables规则：访问service vip+配置的port 也被转发到此端口（从而再转到某pod）。

2）iptables 模式
设置iptables规则：访问service vip+配置的port 直接被转发到后端某pod。此转发由Linux netfilter执行无需切换到用户态所以更高效可靠。
一个缺点是如果选择的第一个pod就不能响应，那么连接会失败（可以使用readiness probes来优化）。

3）IPVS 模式
类似iptables 模式但采用netlink，效率更高。

参考 https://kubernetes.io/docs/concepts/services-networking/service/#the-gory-details-of-virtual-ips

小到中等规模（低于数千service）- user space 模式。超过1万service 时iptables 模式可能性能下降，此时可采用IPVS 模式。

参考 https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

Service types：

• ClusterIP：默认。每个service 在集群内有唯一ip，不能从外部访问
• NodePort：在每个节点的 NodeIP:NodePort 对外暴露该service
• LoadBalancer：云厂商提供，通过一个公网ip 能访问该service
• ExternalName

另外，可通过ingress 来对外暴露服务。

未来service 会有更多的入口方式。

添加k8s3节点

当前有k8s0、k8s1两个节点，部署了helloweb 服务。

按照“记录 - k8s 入门搭建 (1.16.0, helloweb)” 的步骤，添加第三个节点k8s2=192.168.199.202，并加入集群：

[root@k8s0 ~]# kc get node -o wide
NAME   STATUS   ROLES    AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
k8s0   Ready    master   11d     v1.16.0   192.168.199.126   <none>        CentOS Linux 7 (Core)   3.10.0-693.el7.x86_64   docker://18.9.7
k8s1   Ready    <none>   10d     v1.16.0   192.168.199.90    <none>        CentOS Linux 7 (Core)   3.10.0-693.el7.x86_64   docker://18.9.7
k8s2   Ready    <none>   6h46m   v1.16.0   192.168.199.202   <none>        CentOS Linux 7 (Core)   3.10.0-693.el7.x86_64   docker://18.9.7

（注：上面的ip 126|90 是因为虚机一个网卡有多个ip）

修改helloweb

hello_web.go

package main

import (
  "fmt"
  "net/http"
  "os"
)

func main() {
  http.Handle("/", http.HandlerFunc(helloWeb))

  if err := http.ListenAndServe(":8080", nil); err != nil {
    fmt.Println("Server error:", err)
  }
}

func helloWeb(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, "Hello, web!\n")
  name, _ := os.Hostname()
  fmt.Fprintf(w, "Hostname=%s, RemoteAddr=%s\n", name, r.RemoteAddr)
}

增加打印hostname 和请求端地址信息。

按原文方法重新打包为镜像“YYY/helloweb:v0.2” 并上传（“YYY” 替换为你的hub.docker.com 账户名，下同）。

升级部署helloweb

kc edit deploy helloweb
#在编辑区找到“- image: YYY/helloweb”，修改为“- image: YYY/helloweb:v0.2”，保存后退出
#kubernetes会自动更新镜像为v0.2

#如果原来只有1个pod，增加到2个：
kc scale deploy helloweb --replicas=2

NAME                        READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
helloweb-7c899bb846-6sqw4   1/1     Running   3          5h14m   10.100.109.69    k8s2   <none>           <none>
helloweb-7c899bb846-9rpw9   1/1     Running   3          5h31m   10.100.166.221   k8s1   <none>           <none>

#可以看到2个pod分布在两个worker节点上。

#顺便验证NodePort：
kc get svc -o wide

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
helloweb     NodePort    10.96.209.169   <none>        8080:32600/TCP   26h   app=helloweb
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          11d   <none>

#在三个节点均可：
#curl 10.96.209.169:8080
#curl 192.168.199.200:32600 （及201、202）
#注：8080是vip的port，集群内使用。32600是type=NodePort在本机暴露服务的端口，集群外使用

nginx

在k8s2部署nginx。

/etc/nginx/nginx.conf：

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}


http {
    upstream k8s_nodes {
        server 192.168.199.200:32600;
        server 192.168.199.201:32600;
        server 192.168.199.202:32600;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://k8s_nodes;
        }
    }
}

通过浏览器访问“http://192.168.199.202”，结果例如：

Hello, web!
Hostname=helloweb-7c899bb846-6sqw4, RemoteAddr=10.100.150.64:48818

经测试Hostname两个pod均有。RemoteAddr有以下：
10.100.109.64 #k8s2 tunl0 ip
10.100.150.64 #k8s0 tunl0 ip
10.100.166.192 #k8s1 tunl0 ip
192.168.199.90 #k8s1
192.168.199.202 #k8s2

经试验，reboot重启各节点后：svc ip:port 均不变；pod ip变了！（svc ip在其生命期间不变）

手工设置svc nodePort (32600)：

[root@k8s0 ~]# kc edit svc helloweb

在编辑区修改“.spec.ports[0].nodePort” 值，保存并退出后会生效。
注意：修改svc port 8080不担心冲突，因为svc ip (209.169) 是vip，唯一。但修改nodePort 要防止冲突。

安装ingress-nginx

参考 https://kubernetes.github.io/ingress-nginx/deploy/

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
#输出例如：

namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created

#由于是自行安装，选择Bare-metal 方式：
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
#输出：service/ingress-nginx created

#检查安装进度：
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch

NAMESPACE       NAME                                        READY   STATUS             RESTARTS   AGE
ingress-nginx   nginx-ingress-controller-568867bf56-6ls2r   0/1     ImagePullBackOff   0          3m49s

#Ready一直是0，检查日志（-n 指定namespace）：
kc describe pod/nginx-ingress-controller-568867bf56-6ls2r -n ingress-nginx

Name:         nginx-ingress-controller-568867bf56-6ls2r
......
Node:         k8s2/192.168.199.202
......
Containers:
  nginx-ingress-controller:
    Container ID:  
    Image:         quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
......
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  ......
  Warning  Failed     75s                  kubelet, k8s2      Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1": rpc error: code = Unknown desc = dial tcp 54.164.148.166:443: i/o timeout
......

#可见原因是不能拉取quay镜像

#按照 “记录 - k8s 入门搭建 (1.16.0, helloweb)” 所记方法获取镜像，导入k8s2 节点（因为上面显示pod在该节点）
#注：导入后REPOSITORY和TAG可能丢失仅显示“< none >”，可以：
#docker tag 29024c9c6e70 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
#“29024c9c6e70”是image id，由镜像内容唯一确定不会变。
#注：此镜像483MB

#回到k8s0，稍等一会，可看到watch 输出提示成功

[root@k8s0 ~]# kc get deploy -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-ingress-controller   1/1     1            1           24m
[root@k8s0 ~]# kc get pod -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-568867bf56-6ls2r   1/1     Running   0          24m
[root@k8s0 ~]# kc get svc -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.96.162.114   <none>        80:31754/TCP,443:31140/TCP   21m

#（注：经试验即使在ali ECS环境，bare-metal方式安装后也不会产生EXTERNAL-IP）

配置ingress

参考 https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource

ingress-nginx 是一个ingress controller（即具体实现ingress 的路由规则）。具体路由还需配置ingress。

##在k8s0节点
mkdir -p /test/k8s/ingress_nginx
cd /test/k8s/ingress_nginx
vi min.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /testpath
        backend:
          serviceName: helloweb 
          servicePort: 8080

kc apply -f min.yaml
#输出：ingress.networking.k8s.io/test-ingress created

[root@k8s0 ingress_nginx]# kc get svc  -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.96.162.114   <none>        80:31754/TCP,443:31140/TCP   39m
[root@k8s0 ingress_nginx]# 
[root@k8s0 ingress_nginx]# curl 10.96.162.114
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
[root@k8s0 ingress_nginx]# curl 10.96.162.114/testpath
Hello, web!
Hostname=helloweb-7c899bb846-6sqw4, RemoteAddr=10.100.109.70:41538

#Bare-metal采用NodePort
#“curl 10.96.162.114” 返回404 因为min.yaml 里只配置了“/testpath” 的入口规则
#“curl 10.96.162.114/testpath” 在三个节点均可执行

#在浏览器可访问“http://192.168.199.200:31754/testpath”（及201、202）
#输出的RemoteAddr均=10.100.109.70=ing pod ip：

[root@k8s0 ingress_nginx]# kc get pod -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE   READINESS GATES
nginx-ingress-controller-568867bf56-6ls2r   1/1     Running   0          49m   10.100.109.70   k8s2   <none>           <none>

关于ingress

参考 https://github.com/kubernetes/ingress-nginx
及
https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress

Ingress似乎是为了简化将k8s service暴露出去所需的webserver/loadbalancer配置过程。ingress-nginx容器里包含nginx进程：

[root@k8s0 ingress_nginx]# kc exec nginx-ingress-controller-568867bf56-6ls2r -n ingress-nginx -it /bin/sh
$ pwd
/etc/nginx
$ ls
fastcgi_params	geoip  lua  mime.types	modsecurity  modules  nginx.conf  opentracing.json  owasp-modsecurity-crs  template
$ ps -ef|grep nginx
www-data     1     0  0 09:44 ?        00:00:00 /usr/bin/dumb-init -- /nginx-ingress-controller --configmap=ingress-nginx/nginx-configuration --tcp-services-configmap=ingress-nginx/tcp-services --udp-services-configmap=ingress-nginx/udp-services --publish-service=ingress-nginx/ingress-nginx --annotations-prefix=nginx.ingress.kubernetes.io
www-data     6     1  1 09:44 ?        00:00:33 /nginx-ingress-controller --configmap=ingress-nginx/nginx-configuration --tcp-services-configmap=ingress-nginx/tcp-services --udp-services-configmap=ingress-nginx/udp-services --publish-service=ingress-nginx/ingress-nginx --annotations-prefix=nginx.ingress.kubernetes.io
www-data    28     6  0 09:44 ?        00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /etc/nginx/nginx.conf
www-data   115    28  0 09:58 ?        00:00:01 nginx: worker process
www-data   116    28  0 09:58 ?        00:00:02 nginx: worker process
www-data   117    28  0 09:58 ?        00:00:00 nginx: cache manager process
www-data   198   191  0 10:27 pts/0    00:00:00 grep nginx
$ exit

Ingress的性能 (试验中似乎nginx更快)、及其他ingress controller 再研究。

Cleanup

#k8s2 停机

#删除ingress

[root@k8s0 ingress_nginx]# cd /test/k8s/ingress_nginx
[root@k8s0 ingress_nginx]# kc get ingress
NAME           HOSTS   ADDRESS   PORTS   AGE
test-ingress   *                 80      39m
[root@k8s0 ingress_nginx]# kc delete -f min.yaml 
ingress.networking.k8s.io "test-ingress" deleted
[root@k8s0 ingress_nginx]# kc get ingress
No resources found in default namespace.

#删除ingress-nginx svc

[root@k8s0 ingress_nginx]# kc get svc -n ingress-nginx
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.96.162.114   <none>        80:31754/TCP,443:31140/TCP   72m
[root@k8s0 ingress_nginx]# kc delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
service "ingress-nginx" deleted
[root@k8s0 ingress_nginx]# kc get svc -n ingress-nginx
No resources found in ingress-nginx namespace.

#删除ingress-nginx mandatory

[root@k8s0 ingress_nginx]# kc delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
namespace "ingress-nginx" deleted
configmap "nginx-configuration" deleted
configmap "tcp-services" deleted
configmap "udp-services" deleted
serviceaccount "nginx-ingress-serviceaccount" deleted
clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" deleted
role.rbac.authorization.k8s.io "nginx-ingress-role" deleted
rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" deleted
deployment.apps "nginx-ingress-controller" deleted

#命令执行一直不能结束 - 因为pod 所在的k8s2 停机了。

#重启k8s2后，耐心等待5分多钟，命令执行完成。

[root@k8s0 ingress_nginx]# kc get pod -n ingress-nginx
No resources found in ingress-nginx namespace.
[root@k8s0 ingress_nginx]# kc get pod
NAME                        READY   STATUS    RESTARTS   AGE
helloweb-7c899bb846-9rpw9   1/1     Running   3          7h38m
helloweb-7c899bb846-xkt6v   1/1     Running   0          7m21s

#（只剩helloweb）

#注意：强删pod 可能造成重复进程，参考 kc delete --help

Bare-metal的考虑

参考 https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/baremetal.md

TODO