k8s_难产的ingress架构初体验（一）

在 https://editor.csdn.net/md/?articleId=106170316中跟着github中的说明进行了初体验。但是发现并不是每个模块都会用到github说明初学版后端service准备两个version的myapp备用，并配好对应的serviceame:myappsvcNamespace:defaultLabels:<none>Annotations:

酱园里的鸭鸭

5855人浏览 · 2020-05-26 23:08:21

酱园里的鸭鸭 · 2020-05-26 23:08:21 发布

在 https://editor.csdn.net/md/?articleId=106170316中跟着github中的说明进行了初体验。但是发现并不是每个模块都会用到.

github说明

后端service准备

两个version的myapp备用，并配好对应的service

ame:              myappsvc
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myapp
Type:              ClusterIP
IP:                10.96.66.66
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.215:80,10.244.2.216:80,10.244.2.217:80 + 2 more...
Session Affinity:  None
Events:            <none>


Name:              myappsvc2
Namespace:         default
Labels:            <none>
Annotations:       Selector:  app=myappsec
Type:              ClusterIP
IP:                10.96.55.55
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.225:80,10.244.2.226:80,10.244.2.227:80 + 2 more...
Session Affinity:  None

部署ingress.yaml

ingress.yaml 配置

[root@test ~]# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingressmyapp
  namespace: ingress-nginx
spec:
  rules:
  - host: pphqq.vicp.net
    http:
     paths:
     - path: /v1
       backend:
        serviceName: myappsvc
        servicePort: 80
     - path: /v2
       backend:
        serviceName: mappsvc2
        servicePort: 80

apply 以后发现有报错

[root@test ~]# kubectl apply -f ingress.yaml
Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: context deadline exceeded

TS 参考1 ：https://blog.51cto.com/juestnow/2493608

文中提到**# kubelet 参数node-ip 为ipv6 模式记得修改 service ipFamily: IPv6
模式不然新版本webhook 会报错**

没想到什么：看到了 ipFamily: IPv6
https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
提到 !!! danger The admission webhook require conectivity between Kubernetes API server and the ingress controller.
又是一个官方文档 https://kubernetes.io/zh/docs/reference/access-authn-authz/extensible-admission-controllers/
文中提到**对于使用 admissionregistration.k8s.io/v1beta1 创建的 webhook 而言，其默认超时是 30 秒。**这个跟报错中timeout=30s比较像。
自己跑的安装命令：
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/cloud/deploy.yaml
修改这个文件，在service下加了 ipFamily: IPv6 ，apply ingress.yaml 一样有报错。

TS 参考2 https://stackoverflow.com/questions/61365202/nginx-ingress-service-ingress-nginx-controller-admission-not-found

看起来还没解

TS 参考3 https://github.com/kubernetes/ingress-nginx/issues/5583

Orz 这个老哥重装了k8s ，我还能说啥。那研究参考1吧。
先看自己的log

[root@test ~]# kubectl logs pod/ingress-nginx-controller-866488c6d4-2sz9s -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.32.0
  Build:         git-446845114
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.17.10

-------------------------------------------------------------------------------

I0519 14:42:23.085872       6 flags.go:204] Watching for Ingress class: nginx
W0519 14:42:23.086188       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0519 14:42:23.086222       6 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0519 14:42:23.086362       6 main.go:220] Creating API client for https://10.96.0.1:443
I0519 14:42:23.094174       6 main.go:264] Running in Kubernetes cluster version v1.18 (v1.18.2) - git (clean) commit 52c56ce7a8272c798dbc29846288d7cd9fbae032 - platform linux/amd64
I0519 14:42:23.210843       6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0519 14:42:23.213364       6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
W0519 14:42:23.215404       6 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I0519 14:42:23.229793       6 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I0519 14:42:23.260220       6 nginx.go:263] Starting NGINX Ingress controller
I0519 14:42:23.276062       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"a0dec818-3429-4aec-a88a-97adba153201", APIVersion:"v1", ResourceVersion:"4033352", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0519 14:42:24.460640       6 nginx.go:307] Starting NGINX process
I0519 14:42:24.463065       6 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
I0519 14:42:24.463137       6 leaderelection.go:242] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I0519 14:42:24.463627       6 controller.go:139] Configuration changes detected, backend reload required.
I0519 14:42:24.485110       6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0519 14:42:24.485316       6 status.go:86] new leader elected: ingress-nginx-controller-866488c6d4-2sz9s
I0519 14:42:24.537679       6 controller.go:155] Backend successfully reloaded.
I0519 14:42:24.537711       6 controller.go:164] Initial sync, sleeping for 1 second.
E0520 12:28:00.939762       6 leaderelection.go:320] error retrieving resource lock ingress-nginx/ingress-controller-leader-nginx: context deadline exceeded
I0520 12:28:00.939793       6 leaderelection.go:277] failed to renew lease ingress-nginx/ingress-controller-leader-nginx: timed out waiting for the condition
I0520 12:28:00.942992       6 leaderelection.go:242] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I0520 12:28:00.971606       6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx

[root@test ~]# kubectl describe svc ingress-nginx-controller-admission -n  ingress-nginx
Name:              ingress-nginx-controller-admission
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/version=0.32.0
                   helm.sh/chart=ingress-nginx-2.0.3
Annotations:       Selector:  app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:              ClusterIP
IP:                10.101.177.77
Port:              https-webhook  443/TCP
TargetPort:        webhook/TCP
Endpoints:         10.244.2.201:8443
Session Affinity:  None
Events:            <none>

[root@test ~]# curl https://ingress-nginx-controller-admission.ingress-nginx.svc:443
curl: (6) Could not resolve host: ingress-nginx-controller-admission.ingress-nginx.svc; Unknown error

无法解析？
检查 ingress-nginx-controller-admission.ingress-nginx.svc

[root@test ~]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.107.164.64   <pending>     80:31145/TCP,443:30277/TCP   4d18h
ingress-nginx-controller-admission   ClusterIP      10.101.177.77   <none>        443/TCP                      4d18h

[root@test ~]# kubectl describe svc/ingress-nginx-controller-admission -n ingress-nginx
Name:              ingress-nginx-controller-admission
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/version=0.32.0
                   helm.sh/chart=ingress-nginx-2.0.3
Annotations:       Selector:  app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:              ClusterIP
IP:                10.101.177.77
Port:              https-webhook  443/TCP
TargetPort:        webhook/TCP
Endpoints:         10.244.2.201:8443
Session Affinity:  None
Events:            <none>
[root@test ~]#

太艰难了 5/25，上班忙到炸。 ingress已经折磨我好久了，这篇已经磨了一礼拜。过生日前让我release这篇吧。TAT
dns有问题，看一眼服务吧

[root@test ~]# kubectl get svc -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   26d
# 把dns改了
[root@test ~]# vim /etc/resolv.conf
[root@test ~]# nslookup baidu.com
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
Name:   baidu.com
Address: 39.156.69.79
Name:   baidu.com
Address: 220.181.38.148

还是没用
又看到一篇：https://blog.csdn.net/u012986012/article/details/105453459
还是提到了webhook的问题
回到官网 https://kubernetes.io/zh/docs/reference/access-authn-authz/extensible-admission-controllers/
其中提到了失败策略
failurePolicy 定义了如何处理 admission webhook 中无法识别的错误和超时错误。允许的值为 Ignore 或 Fail。

Ignore 表示调用 webhook 的错误将被忽略并且允许 API 请求继续。
Fail 表示调用 webhook 的错误导致准入失败并且 API 请求被拒绝。

试试ignore吧或者关掉webhook?

在本片最顶上的一张图有两个关于webhook的资源
一个是https_webhook, 另一个是validatingWebhookConfiguration

即时配置 admission webhook
您可以通过 ValidatingWebhookConfiguration 或者 MutatingWebhookConfiguration 动态配置哪些资源要被哪些 admission webhook 处理。

好像找对路子了。
0526 今天deadline
先关掉试试
用edit 删除这三行
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
没啥用，不懂要怎么关，各位大神麻烦评论告诉我
加ignore试试
也是用edit

[root@test ~]# kubectl get ValidatingWebhookConfiguration/ingress-nginx-admission -n ingress-nginx
NAME                      WEBHOOKS   AGE
ingress-nginx-admission   1          7d1h
[root@test ~]# kubectl edit ValidatingWebhookConfiguration/ingress-nginx-admission -n ingress-nginx
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission edited
######下面是edit界面中的某一段
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
  caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJPRENCM3FBREFnRUNBaEJOUUROWnBJcXR3TTJzaDg3YVVrSlpNQW9HQ0NxR1NNNDlCQU1DTUFBd0lCY04KTWpBd05URTVNVEl3TlRJMVdoZ1BNakV5TURBME1qVXhNakExTWpWYU1BQXdXVEFUQmdjcWhrak9QUUlCQmdncQpoa2pPUFFNQkJ3TkNBQVRDcWRWOWdVTm5yelY0Umg5NlIxYkl6R3lzQjE3UHJzc21CNDBGQjRPYU5MR28yMGwzCjJHYVpaTHNtN1dzblJva2FRcWpMOSs3NlBUQVBRSm9KS3BrYW96Z3dOakFPQmdOVkhROEJBZjhFQkFNQ0FnUXcKRXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdFd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBS0JnZ3Foa2pPUFFRRApBZ05KQURCR0FpRUE4K1A1WGlyOFgzREJEK1IvN2lIVWxzNXg3TW1UdjVMN1VzQzBJM3N2WmM4Q0lRQ2JSMUNpCkVOdEJ6N21CNU9KWG50RmdDblJLRkxjUVkxeWJ6bDVwTnlyZkZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  service:
    name: ingress-nginx-controller-admission
    namespace: ingress-nginx
    path: /extensions/v1beta1/ingresses
    port: 443
failurePolicy: Fail             ##################改成Ignore
matchPolicy: Exact
name: validate.nginx.ingress.kubernetes.io
#########################然后重新部署ingress.yaml
[root@test ~]# kubectl apply -f ingress.yaml
ingress.extensions/ingressmyapp created   
[root@test ~]#

TAT 谢天谢地。
但是照理说Fail不该动的，先欠着吧，这篇只是初体验。XD

验证篇

[root@test ~]# kubectl get ingress -n ingress-nginx
NAME           CLASS    HOSTS            ADDRESS   PORTS   AGE
ingressmyapp   <none>   pphqq.vicp.net             80      11m
########在describe时发现了error ####################################
[root@test ~]# kubectl describe ingress/ingressmyapp -n ingress-nginx
Name:             ingressmyapp
Namespace:        ingress-nginx
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
###########################为什么会找不到？ 
Rules:
Host            Path  Backends
----            ----  --------
pphqq.vicp.net
                /v1   myappsvc:80 (<error: endpoints "myappsvc" not found>)   
                ##    **所以必须在一个ns里？ ** 改完一个以后这个变成   /v1   myappsvc:80 (<none>)
                /v2   mappsvc2:80 (<error: endpoints "mappsvc2" not found>)
                #    v2是写错服务名，重新改
Annotations:      Events:
Type            Reason  Age   From                      Message
----            ------  ----  ----                      -------
Normal          CREATE  12m   nginx-ingress-controller  Ingress ingress-nginx/ingressmyapp

改完再看describe，现在看起来都已经有了

[root@test ~]# kubectl describe ingress/ingressmyapp -n ingress-nginx
Name:             ingressmyapp
Namespace:        ingress-nginx
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

###########################为什么会找不到？
#官网 Note: Depending on the Ingress controller you are using, you may need to create a default-http-backend Service.
Default Backend
An Ingress with no rules sends all traffic to a single default backend. The default backend is typically a configuration option of the Ingress controller and is not specified in your Ingress resources.

If none of the hosts or paths match the HTTP request in the Ingress objects, the traffic is routed to your default backend.
所以应该不影响。只有再不匹配路径是会转到默认。

Rules:
  Host            Path  Backends
  ----            ----  --------
  pphqq.vicp.net
                  /v1   myappsvc:80 (10.244.2.232:80,10.244.2.235:80,10.244.2.239:80 + 2 more...)
                  /v2   myappsvc2:80 (10.244.2.231:80,10.244.2.236:80,10.244.2.237:80 + 2 more...)
Annotations:      Events:
  Type            Reason  Age    From                      Message
  ----            ------  ----   ----                      -------
  Normal          CREATE  41m    nginx-ingress-controller  Ingress ingress-nginx/ingressmyapp
  Normal          CREATE  7m10s  nginx-ingress-controller  Ingress ingress-nginx/ingressmyapp
  Normal          UPDATE  77s    nginx-ingress-controller  Ingress ingress-nginx/ingressmyapp

验证放在第二篇吧。

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub