kubernetes1.22 helm ingress-nginx 使用笔记

itxiaoqCheng

2070人浏览 · 2022-09-26 11:41:40

itxiaoqCheng · 2022-09-26 11:41:40 发布

Ingress 是对集群中服务的外部访问进行管理的 API 对象，典型的访问方式是 HTTP。

client -> domain 解析到 -> svc ingress-nginx-controller -> pod ingress-nginx-controller -> ns svc(endpoints) -> pod

# helm list

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION

ingress-nginx default 1 2022-09-22 22:49:39.476083927 +0800 CST deployed ingress-nginx-4.2.5 1.3.1

把已经安装的ingress-nginx先删掉

# helm uninstall ingress-nginx

release "ingress-nginx" uninstalled

删掉helm repo

# helm repo list

NAME URL

grafana https://grafana.github.io/helm-charts

ingress-nginx https://kubernetes.github.io/ingress-nginx

# helm repo remove ingress-nginx

"ingress-nginx" has been removed from your repositories

添加到repo并更新repo

# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

"ingress-nginx" has been added to your repositories

# helm repo update

Hang tight while we grab the latest from your chart repositories...

...Successfully got an update from the "ingress-nginx" chart repository

...Successfully got an update from the "grafana" chart repository

Update Complete. ⎈Happy Helming!⎈

解压并安装

root@VM-0-2-ubuntu:/data/cloud/ingress# helm fetch ingress-nginx/ingress-nginx

root@VM-0-2-ubuntu:/data/cloud/ingress#

root@VM-0-2-ubuntu:/data/cloud/ingress# tar -xvf ingress-nginx-4.2.5.tgz && cd ingress-nginx/

ingress-nginx/Chart.yaml

ingress-nginx/values.yaml

ingress-nginx/templates/NOTES.txt

ingress-nginx/templates/_helpers.tpl

ingress-nginx/templates/_params.tpl

ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml

ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml

ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml

ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml

ingress-nginx/templates/admission-webhooks/job-patch/psp.yaml

ingress-nginx/templates/admission-webhooks/job-patch/role.yaml

ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml

ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml

ingress-nginx/templates/admission-webhooks/validating-webhook.yaml

ingress-nginx/templates/clusterrole.yaml

ingress-nginx/templates/clusterrolebinding.yaml

ingress-nginx/templates/controller-configmap-addheaders.yaml

ingress-nginx/templates/controller-configmap-proxyheaders.yaml

ingress-nginx/templates/controller-configmap-tcp.yaml

ingress-nginx/templates/controller-configmap-udp.yaml

ingress-nginx/templates/controller-configmap.yaml

ingress-nginx/templates/controller-daemonset.yaml

ingress-nginx/templates/controller-deployment.yaml

ingress-nginx/templates/controller-hpa.yaml

ingress-nginx/templates/controller-ingressclass.yaml

ingress-nginx/templates/controller-keda.yaml

ingress-nginx/templates/controller-poddisruptionbudget.yaml

ingress-nginx/templates/controller-prometheusrules.yaml

ingress-nginx/templates/controller-psp.yaml

ingress-nginx/templates/controller-role.yaml

ingress-nginx/templates/controller-rolebinding.yaml

ingress-nginx/templates/controller-service-internal.yaml

ingress-nginx/templates/controller-service-metrics.yaml

ingress-nginx/templates/controller-service-webhook.yaml

ingress-nginx/templates/controller-service.yaml

ingress-nginx/templates/controller-serviceaccount.yaml

ingress-nginx/templates/controller-servicemonitor.yaml

ingress-nginx/templates/controller-wehbooks-networkpolicy.yaml

ingress-nginx/templates/default-backend-deployment.yaml

ingress-nginx/templates/default-backend-hpa.yaml

ingress-nginx/templates/default-backend-poddisruptionbudget.yaml

ingress-nginx/templates/default-backend-psp.yaml

ingress-nginx/templates/default-backend-role.yaml

ingress-nginx/templates/default-backend-rolebinding.yaml

ingress-nginx/templates/default-backend-service.yaml

ingress-nginx/templates/default-backend-serviceaccount.yaml

ingress-nginx/templates/dh-param-secret.yaml

ingress-nginx/.helmignore

ingress-nginx/CHANGELOG.md

ingress-nginx/OWNERS

ingress-nginx/README.md

ingress-nginx/README.md.gotmpl

ingress-nginx/ci/controller-custom-ingressclass-flags.yaml

ingress-nginx/ci/daemonset-customconfig-values.yaml

ingress-nginx/ci/daemonset-customnodeport-values.yaml

ingress-nginx/ci/daemonset-extra-modules.yaml

ingress-nginx/ci/daemonset-headers-values.yaml

ingress-nginx/ci/daemonset-internal-lb-values.yaml

ingress-nginx/ci/daemonset-nodeport-values.yaml

ingress-nginx/ci/daemonset-podannotations-values.yaml

ingress-nginx/ci/daemonset-tcp-udp-configMapNamespace-values.yaml

ingress-nginx/ci/daemonset-tcp-udp-portNamePrefix-values.yaml

ingress-nginx/ci/daemonset-tcp-udp-values.yaml

ingress-nginx/ci/daemonset-tcp-values.yaml

ingress-nginx/ci/deamonset-default-values.yaml

ingress-nginx/ci/deamonset-metrics-values.yaml

ingress-nginx/ci/deamonset-psp-values.yaml

ingress-nginx/ci/deamonset-webhook-and-psp-values.yaml

ingress-nginx/ci/deamonset-webhook-values.yaml

ingress-nginx/ci/deployment-autoscaling-behavior-values.yaml

ingress-nginx/ci/deployment-autoscaling-values.yaml

ingress-nginx/ci/deployment-customconfig-values.yaml

ingress-nginx/ci/deployment-customnodeport-values.yaml

ingress-nginx/ci/deployment-default-values.yaml

ingress-nginx/ci/deployment-extra-modules.yaml

ingress-nginx/ci/deployment-headers-values.yaml

ingress-nginx/ci/deployment-internal-lb-values.yaml

ingress-nginx/ci/deployment-metrics-values.yaml

ingress-nginx/ci/deployment-nodeport-values.yaml

ingress-nginx/ci/deployment-podannotations-values.yaml

ingress-nginx/ci/deployment-psp-values.yaml

ingress-nginx/ci/deployment-tcp-udp-configMapNamespace-values.yaml

ingress-nginx/ci/deployment-tcp-udp-portNamePrefix-values.yaml

ingress-nginx/ci/deployment-tcp-udp-values.yaml

ingress-nginx/ci/deployment-tcp-values.yaml

ingress-nginx/ci/deployment-webhook-and-psp-values.yaml

ingress-nginx/ci/deployment-webhook-extraEnvs-values.yaml

ingress-nginx/ci/deployment-webhook-resources-values.yaml

ingress-nginx/ci/deployment-webhook-values.yaml

检查主节点污点taint，要与daemonset-prod.yaml里面的一直，我这里安装的时候去掉了，所以把容忍度（tolerations）注释掉

root@VM-0-2-ubuntu:~# kubectl describe no vm-0-2-ubuntu

Name: vm-0-2-ubuntu

Roles: control-plane,master

Labels: beta.kubernetes.io/arch=amd64

beta.kubernetes.io/os=linux

ingress=true

kubernetes.io/arch=amd64

kubernetes.io/hostname=vm-0-2-ubuntu

kubernetes.io/os=linux

node-role.kubernetes.io/control-plane=

node-role.kubernetes.io/master=

node.kubernetes.io/exclude-from-external-load-balancers=

Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock

node.alpha.kubernetes.io/ttl: 0

projectcalico.org/IPv4Address: 172.29.0.2/20

projectcalico.org/IPv4IPIPTunnelAddr: 10.244.171.128

volumes.kubernetes.io/controller-managed-attach-detach: true

CreationTimestamp: Fri, 16 Sep 2022 14:08:31 +0800

Taints: <none>

Unschedulable: false

Lease:

HolderIdentity: vm-0-2-ubuntu

AcquireTime: <unset>

RenewTime: Mon, 26 Sep 2022 10:28:20 +0800

创建 vim daemonset-prod.yaml 文件

需要注意nodeSelector和tolerations的值

# ci/daemonset-prod.yaml

controller:

name: controller

image:

repository: cnych/ingress-nginx

tag: "v1.1.0"

digest:

dnsPolicy: ClusterFirstWithHostNet

hostNetwork: true

publishService: # hostNetwork 模式下设置为false，通过节点IP地址上报ingress status数据

enabled: false

# 是否需要处理不带 ingressClass 注解或者 ingressClassName 属性的 Ingress 对象

# 设置为 true 会在控制器启动参数中新增一个 --watch-ingress-without-class 标注

watchIngressWithoutClass: false

kind: DaemonSet

#tolerations: # kubeadm 安装的集群默认情况下master是有污点，需要容忍这个污点才可以部署

#- key: "node-role.kubernetes.io/master"

# operator: "Equal"

# effect: "NoSchedule"

nodeSelector: # 固定到master节点

kubernetes.io/hostname: vm-0-2-ubuntu

service: # HostNetwork 模式不需要创建service

enabled: false

admissionWebhooks: # 强烈建议开启 admission webhook

enabled: true

createSecretJob:

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

patchWebhookJob:

resources:

limits:

cpu: 10m

memory: 20Mi

requests:

cpu: 10m

memory: 20Mi

patch:

enabled: true

image:

repository: cnych/ingress-nginx-webhook-certgen

tag: v1.1.1

digest:

defaultBackend: # 配置默认后端

enabled: true

name: defaultbackend

image:

repository: cnych/ingress-nginx-defaultbackend

tag: "1.5"

创建namespace

# kubectl create ns ingress-nginx

namespace/ingress-nginx created

安装

root@VM-0-2-ubuntu:/data/cloud/ingress/ingress-nginx# helm upgrade --install ingress-nginx . -f ./ci/daemonset-prod.yaml --namespace ingress-nginx

Release "ingress-nginx" does not exist. Installing it now.

NAME: ingress-nginx

LAST DEPLOYED: Mon Sep 26 10:20:06 2022

NAMESPACE: ingress-nginx

STATUS: deployed

REVISION: 1

TEST SUITE: None

NOTES:

The ingress-nginx controller has been installed.

It may take a few minutes for the LoadBalancer IP to be available.

You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: example

namespace: foo

spec:

ingressClassName: nginx

rules:

- host: www.example.com

http:

paths:

- pathType: Prefix

backend:

service:

name: exampleService

port:

number: 80

path: /

# This section is only required if TLS is to be enabled for the Ingress

tls:

- hosts:

- www.example.com

secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

apiVersion: v1

kind: Secret

metadata:

name: example-tls

namespace: foo

data:

tls.crt: <base64 encoded cert>

tls.key: <base64 encoded key>

type: kubernetes.io/tls

检查ingress-controller和defaultbackend

root@VM-0-2-ubuntu:/data/cloud/ingress/ingress-nginx# kubectl get svc -n ingress-nginx

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

ingress-nginx-controller-admission ClusterIP 10.100.194.103 <none> 443/TCP 34s

ingress-nginx-defaultbackend ClusterIP 10.100.131.61 <none> 80/TCP 34s

root@VM-0-2-ubuntu:~# kubectl get pods -n ingress-nginx

NAME READY STATUS RESTARTS AGE

ingress-nginx-controller-g9kvw 1/1 Running 0 106s

ingress-nginx-defaultbackend-84854cd6cb-m9w4c 1/1 Running 0 13m

root@VM-0-2-ubuntu:~# kubectl logs -f ingress-nginx-controller-g9kvw -n ingress-nginx

W0926 02:32:24.207948 7 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

I0926 02:32:24.208077 7 main.go:223] "Creating API client" host="https://10.96.0.1:443"

I0926 02:32:24.222865 7 main.go:267] "Running in Kubernetes cluster" major="1" minor="22" git="v1.22.2" state="clean" commit="8b5a19147530eaac9476b0ab82980b4088bbc1b2" platform="linux/amd64"

-------------------------------------------------------------------------------

NGINX Ingress controller

Release: v1.1.0

Build: cacbee86b6ccc45bde8ffc184521bed3022e7dee

Repository: https://github.com/kubernetes/ingress-nginx

nginx version: nginx/1.19.9

-------------------------------------------------------------------------------

I0926 02:32:24.227436 7 main.go:86] "Valid default backend" service="ingress-nginx/ingress-nginx-defaultbackend"

I0926 02:32:24.317841 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"

I0926 02:32:24.332754 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"

I0926 02:32:24.354920 7 nginx.go:255] "Starting NGINX Ingress controller"

I0926 02:32:24.364860 7 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"4ffb51bc-8101-4a6b-a4bb-9093e0e6440c", APIVersion:"v1", ResourceVersion:"2229530", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller

I0926 02:32:25.557677 7 nginx.go:297] "Starting NGINX process"

I0926 02:32:25.557884 7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...

I0926 02:32:25.558142 7 nginx.go:317] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"

I0926 02:32:25.558550 7 controller.go:155] "Configuration changes detected, backend reload required"

I0926 02:32:25.574133 7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader

I0926 02:32:25.574633 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-g9kvw"

I0926 02:32:25.591606 7 status.go:215] "POD is not ready" pod="ingress-nginx/ingress-nginx-controller-g9kvw" node="vm-0-2-ubuntu"

I0926 02:32:25.625866 7 controller.go:172] "Backend successfully reloaded"

I0926 02:32:25.626086 7 controller.go:183] "Initial sync, sleeping for 1 second"

I0926 02:32:25.626172 7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-g9kvw", UID:"83fe4951-4822-4de3-ab45-9c6636fa8619", APIVersion:"v1", ResourceVersion:"2231263", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

2022/09/26 02:46:14 [crit] 32#32: *6818 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 64.62.197.167, server: 0.0.0.0:443

443报错不管，因为没有装ssl

root@VM-0-2-ubuntu:~# kubectl get ingressclass

NAME CONTROLLER PARAMETERS AGE

nginx k8s.io/ingress-nginx <none> 31m

部署deployment、service、ingress对象

root@VM-0-2-ubuntu:/data/cloud/ingress# cat nginx-deploy.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

name: nginx-deployment

spec:

replicas: 1

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx

ports:

- containerPort: 80

root@VM-0-2-ubuntu:/data/cloud/ingress# cat service.yaml

apiVersion: v1

kind: Service

metadata:

name: nginx-basic

spec:

type: ClusterIP

ports:

- port: 80

targetPort: 80

protocol: TCP

name: http

selector:

app: nginx

root@VM-0-2-ubuntu:/data/cloud/ingress# cat ingress.yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: test-ingress

namespace: default

annotations:

nginx.ingress.kubernetes.io/rewrite-target: /

kubernetes.io/ingress.class: "nginx"

spec:

rules: #定义转发规则

- host: www.ingress.com #指定域名方式

   http:

paths:

- path: / #指定访问的路径

pathType: Prefix #定义路径的类型

backend: #定义转发后端的服务

   service: #定义转发的service

   name: nginx-basic

   port:

number: 80 #由于Ingress控制器开启了hostNetwork: true。这里必须设置80

修改etc/hosts文件

root@VM-0-2-ubuntu:/data/cloud/ingress# cat /etc/hosts

#

127.0.1.1 localhost.localdomain VM-0-2-ubuntu

127.0.0.1 localhost

172.29.0.2 localhost

172.29.0.2 www.ingress.com

测试访问curl -v www.ingress.com

root@VM-0-2-ubuntu:/data/cloud/ingress# curl -v www.ingress.com

* Trying 172.29.0.2:80...

* TCP_NODELAY set

* Connected to www.ingress.com (172.29.0.2) port 80 (#0)

> GET / HTTP/1.1

> Host: www.ingress.com

> User-Agent: curl/7.68.0

> Accept: */*

>

* Mark bundle as not supporting multiuse

< HTTP/1.1 200 OK

< Date: Mon, 26 Sep 2022 03:39:44 GMT

< Content-Type: text/html

< Content-Length: 615

< Connection: keep-alive

< Last-Modified: Tue, 28 Dec 2021 15:28:38 GMT

< ETag: "61cb2d26-267"

< Accept-Ranges: bytes

<

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>

html { color-scheme: light dark; }

body { width: 35em; margin: 0 auto;

font-family: Tahoma, Verdana, Arial, sans-serif; }

</style>

</head>

<body>

<h1>Welcome to nginx!</h1>

<p>If you see this page, the nginx web server is successfully installed and

working. Further configuration is required.</p>

<p>For online documentation and support please refer to

<a href="http://nginx.org/">nginx.org</a>.<br/>

Commercial support is available at

<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

</body>

</html>

* Connection #0 to host www.ingress.com left intact

部署完成。

Cloudpods

开源、云原生的融合云平台

更多推荐

面向未来的 IT 基础设施管理架构——融合云（Unified IaaS）

随着数字化时代的到来，IT系统已成为人类社会正常运转不可或缺的组成部分。不远的未来，智能制造，5G和人工智能等技术将成为推动生产力发展的重要引擎，人类社会将面临前所未有的全面彻底的数字化浪潮。IT基础设施作为IT系统运行的平台和载体，是实现数字化的基石。在这场数字化浪潮中，企业必须积极拥抱云计算技术，采用符合技术发展趋势、面向未来的IT基础构架，才能在未来的竞争中赢得先机。一、云计算历经十余年

Cloudpods

Cloudpods负载均衡的功能介绍

作者:周有松今天的内容会从以下几个方面展开：负载均衡产品简介。主要介绍负载均衡作为一个云上产品，它的功能模型是怎样的，日常使用中会遇到的业务词汇负载均衡的功能与典型应用场景。这部分主要结合业务词汇，对负载均衡服务中常见的一些功能选项进行介绍，并举例介绍一些典型的应用场景最后，我们做一下总结，讨论一下负载均衡产品相比传统方式的优点一、产品简介 1. 以NGINX为例提到负载均衡，我们以

Cloudpods

使用Linux vfio将Nvidia GPU透传给QEMU虚拟机

Linux 上虚拟机 GPU 透传需要使用 vfio 的方式。主要是因为在 vfio 方式下对虚拟设备的权限和 DMA 隔离上做的更好。但是这么做也有个缺点，这个物理设备在主机和其他虚拟机都不能使用了。 qemu 直接使用物理设备本身命令行是很简单的，关键在于事先在主机上对系统、内核和物理设备的一些配置。单纯从 qemu 的命令行来看，其实和普通虚拟机启动就差了最后那个-device的选项。这