k8s部署实例
k8s yaml 实例
·
k8s部署实例
1. 持久化nginx或php等静态资源的pod
1.1 需求
不想每次都把代码打包到nginx镜像，想要持久化代码，但是持久化代码需要挂载网络磁盘nfs或oss等，每次开发完成，都需要将代码放到网络磁盘中，不想每次都手动更新网络磁盘中的代码。
1.2 方案
(一个pod内同时有nginx+php容器也可以这么做，多个容器同理)
设置一个初始化容器（初始化容器用过之后就会消失），初始化容器挂载nfs到指定目录，然后利用git拉取代码到指定目录，这样nfs中就有代码了
因为pod启动时会先启动初始化容器（详情见：https://blog.csdn.net/martinlinux/article/details/125299226 >> 2.1 pod过程），所以在将nginx容器挂载上nfs，这样nginx的html目录中就有代码了
1.3 解决步骤
- 创建git拉取时使用的账号密码 secret资源
我这里的账号密码是root 12345678，因为使用Opaque类型的secret，所以需要先把账号密码加密
echo ‘root’ | base64
echo ‘12345678’ | base64
vim git-base.yaml
apiVersion: v1
kind: Secret
metadata:
name: git-secret
namespace: dev
type: Opaque
data:
username: cm9vdA==
password: MTIzNDU2Nzg=
- 部署pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-web
labels:
app: nginx-web
env: dev
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
app: nginx-web
template:
metadata:
labels:
app: nginx-web
spec:
initContainers:
- name: git-base
image: git-base:2.24.4
env: #因为初始化容器是一次性的，所以这里直接把账号密码映射到环境变量
- name: GIT_USER
valueFrom:
secretKeyRef:
name: git-secret
key: username
- name: GIT_PASSWORD
valueFrom:
secretKeyRef:
name: git-secret
key: password
workingDir: /usr/local/src
command: ['/bin/sh','-c']
args: ['git clone http://$GIT_USER:$GIT_PASSWORD@192.168.8.10/prd_web/web01.git']
volumeMounts:
- mountPath: /usr/local/src
name: htmldata
containers:
- name: nginx-web
image: nginx-base:1.16.1
volumeMounts:
- mountPath: /usr/local/nginx/html
name: htmldata
volumes:
- name: htmldata
nfs:
server: 192.168.8.30
path: /data/redis
readOnly: false
---
apiVersion: v1
kind: Service
metadata:
name: nginx-web-svc
namespace: dev
spec:
type: NodePort
selector:
app: nginx-web
ports:
- name: nginx-http
protocol: TCP
port: 80
targetPort: 80
2. 持久化nginx+php等静态资源的pod（OSS方式）
2.1 需求
不想每次都把代码打包到nginx镜像和php镜像，(配置文件也需要实时更新)想要持久化代码，但是持久化代码需要挂载网络磁盘nfs或oss等，每次开发完成，都需要将代码放到网络磁盘中，不想每次都手动更新网络磁盘中的代码。
2.2 方案
(一个pod内同时有nginx+php容器也可以这么做，多个容器同理)
设置一个初始化容器（初始化容器用过之后就会消失），初始化容器挂载nfs到指定目录，然后利用git拉取代码到指定目录，这样nfs中就有代码了
因为pod启动时会先启动初始化容器（详情见：https://blog.csdn.net/martinlinux/article/details/125299226 >> 2.1 pod过程），所以在将nginx容器挂载上nfs，这样nginx的html目录中就有代码了
2.3 解决步骤
2.3.1 configmap资源
1. 创建拉取git凭证(这里是使用ssh方式拉取的)
注意： 和下方secret资源中的git凭证是一样的git-secret，无状态部署的时候使用任何一个都可以
apiVersion: v1
kind: ConfigMap
metadata:
name: tdd-code-configmap
namespace: testphp
data:
id_rsa: |-
-----BEGIN RSA PRIVATE KEY-----
私钥
-----END RSA PRIVATE KEY-----
id_rsa.pub: >-
公钥
#创建这个是因为拉取的时候需要输入yes
known_hosts: >-
数据
2.nginx+php服务配置文件
apiVersion: v1
kind: ConfigMap
metadata:
name: tdd-web-configmap
namespace: testphp
data:
nginx.conf: |-
user root;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name 0.0.0.0;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html/cka_test;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ \.php$ {
root html/cka_test;
###因为部署在一个pod之中所以写127.0.0.1或者svc的名称都可以
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
include fastcgi_params;
}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
}
php-fpm.conf: |-
[global]
pid = /usr/local/var/run/php-fpm.pid
error_log = /usr/local/var/log/php/php-fpm.log
daemonize = no
include=etc/php-fpm.d/*.conf
www.conf: |-
[www]
user = root
group = root
listen = 127.0.0.1:9000
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
pm.max_requests = 1000
slowlog = /usr/local/var/log/$pool.log.slow
request_slowlog_timeout = 15
request_slowlog_trace_depth = 20
2.4 secret资源
1.创建拉取git的凭证
apiVersion: v1
kind: Secret
metadata:
name: git-secret
namespace: testphp
type: Opaque
data:
#这些是base64转码过的
id_rsa: >-
blUvVnpKYU9GU2xlU29yaXdmdGN4WStZcjlVUENyZ21tNFZCQ1FSM3gwSHRIN3lhT00KM1RkNjB5ZEV4cmVNNzJzZVc5elFDQ1h5UDl1aHUyVnBKNS9nWlFJREFRQUJBb0lCQVFDMml1dnVuNlNUMWovagptWjlYditmUnZHZ1k3MU5OS2FoTVNMZmNVNnRIcWVVUE5Wby9Fc3pxRXl4c0tHcWZMeEE1cU9rL21FbHM1Qmw0Cjh0cmphMC9KR3FSTHFwRzh3cXJwUzJjcGk0SzdVeVAyTW1lcUFXZ09kMnZzbDhmcEpvVmdoVmwwT0VQSlpRbXUKNWpCMjhrTlNsbzJ3djZyWmMzV1FKWTl4VmJhQWVER2cyak1NWVkzUFJqQUFYd3FpU2Ftd3FCL210TEsyaGdUUApRUXVQcjNYZzhwRjZDbjRBVGhyWWtjeGNxckIzWWYwYkcvaGJGTFI3RTlxSFdETUZ1SGlrS3BUS000OEgweWIxCjA1WFN5cW5rVVVMYnZ1NFhkT3Y1VENDMVdLeDhUbmdt
id_rsa_pub: >-
VLaXVMQisxekZqNWl2MVE4S3VDYWJoVUVKQkhmSFFlMGZ2Sm80emROM3JUSjBUR3Q0enZheDViM05BSUpmSS8yNkc3Wldrbm4rQmwgcm9vdEBtYXN0ZXIwMQ==
known_hosts: >-
E2NS41yRUFBQUFEQVFBQkFBQUJBUUQ2V2JWYWxjelNES0M0a3RqUnpETkMxcUlOREFWaXcvb0dBOHVkN3FINUY5WEZaNFFBV2habUZCc3ArY3BwQklPTTYzZXhqYzRPOXZiek1yZDdBVUl0WGlneW1HZlRxckd0NVZ5a0lpYU5oan
2.创建读写oss凭证
apiVersion: v1
data:
#阿里云账号的AKID 需要读写权限
akId: AccessKey ID
akSecret: AccessKey Secret
kind: Secret
metadata:
name: tdd-oss-secret
namespace: testphp
type: Opaque
3.创建拉取容器镜像仓库凭证
- 找一台机器登录docker longin登录仓库 docker login url
- base64 -w 0 ~/.docker/config.json 生成的数据写入下方的data中
apiVersion: v1
data:
.dockerconfigjson: 上方生成的数据
kind: Secret
metadata:
name: docker-secret
namespace: testphp
type: kubernetes.io/dockerconfigjson
2.5 存储
2.5.1 代码存储卷OSS-PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: tdd-web-pv
labels:
alicloud-pvname: tdd-web-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 20Gi
##选择csi
csi:
driver: ossplugin.csi.alibabacloud.com
nodePublishSecretRef:
#选择testphp命名空间下的保密字典test-toodudu
name: test-toodudu
namespace: testphp
volumeAttributes:
bucket: test-k8s-toodudu
otherOpts: ''
url: oss-url.aliyuncs.com
volumeHandle: tdd-web-pv
persistentVolumeReclaimPolicy: Retain
storageClassName: oss
volumeMode: Filesystem
2.5.2 代码存储声明OSS-PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tdd-web-pvc
namespace: testphp
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
limits:
storage: 5Gi
volumeName: tdd-web-pv
volumeMode: Filesystem
storageClassName: oss
2.5.3 日志存储卷OSS-LOG-PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: tdd-web-log-pv
labels:
alicloud-pvname: tdd-web-log-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 60Gi
##选择csi
csi:
driver: ossplugin.csi.alibabacloud.com
nodePublishSecretRef:
#选择testphp命名空间下的保密字典test-toodudu
name: test-toodudu
namespace: testphp
volumeAttributes:
bucket: test-k8s-toodudu-log
otherOpts: ''
url: oss-url.aliyuncs.com
volumeHandle: tdd-web-log-pv
persistentVolumeReclaimPolicy: Retain
storageClassName: oss
volumeMode: Filesystem
2.5.4 日志存储卷OSS-LOG-PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tdd-web-log-pvc
namespace: testphp
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
limits:
storage: 50Gi
volumeName: tdd-web-log-pv
volumeMode: Filesystem
storageClassName: oss
2.6 无状态资源
apiVersion: apps/v1
kind: Deployment
metadata:
name: tdd-web
labels:
app: tdd-web
env: testphp
namespace: testphp
spec:
replicas: 1
selector:
matchLabels:
app: tdd-web
template:
metadata:
labels:
app: tdd-web
spec:
imagePullSecrets:
- name: docker-secret
#初始化容器
initContainers:
- name: git-base
image: registry.cn-beijing.aliyuncs.com/ygbid-docker/base:git-2.24.4
imagePullPolicy: Always
env:
- name: GIT_URL
value: 'git@codeup.aliyun.com:5eb94095053c10a2a600128a/ibi_yunwei/cka_test.git'
- name: GIT_BRANCH
value: 'master'
- name: id_rsa
valueFrom:
#这里我使用的是secret资源中git凭证，使用configmap资源中的git凭证也可以，使用configmap的时候参数是需要改变的
secretKeyRef:
name: git-secret
key: id_rsa
- name: id_rsa_pub
valueFrom:
secretKeyRef:
name: git-secret
key: id_rsa_pub
- name: known_hosts
valueFrom:
secretKeyRef:
name: git-secret
key: known_hosts
workingDir: /usr/local/src
#将git拉取凭证写入ssh密钥文件中，然后拉取代码
command: ['/bin/sh','-c']
args: ['echo -e "$id_rsa" > /root/.ssh/id_rsa && echo $id_rsa_pub > /root/.ssh/id_rsa.pub && echo $known_hosts > /root/.ssh/known_hosts && chmod 0600 /root/.ssh/* && rm -rf ./cka_test && git clone -b $GIT_BRANCH $GIT_URL']
volumeMounts:
- mountPath: /usr/local/src
name: tdd-web-data
containers:
- name: nginx-web
image: registry.cn-beijing.aliyuncs.com/ygbid-docker/base:nginx-1.16.1
imagePullPolicy: Always
volumeMounts:
#挂载代码卷
- mountPath: /usr/local/nginx/html
name: tdd-web-data
#挂载配置卷
- name: tdd-web-configmap
mountPath: /usr/local/nginx/conf/nginx.conf
subPath: nginx.conf
#挂载日志卷
- name: tdd-log
mountPath: /usr/local/nginx/logs
- name: php-web
image: registry.cn-beijing.aliyuncs.com/ygbid-docker/base:php-7.3.8
imagePullPolicy: Always
#因为我配置文件中是root启动，所以这里需要改一下php的启动命令，-R表示允许root启动
command: ['php-fpm','-R']
volumeMounts:
#挂载代码卷
- mountPath: /var/www/html
name: tdd-web-data
#挂载配置卷
- name: tdd-web-configmap
mountPath: /usr/local/etc/php-fpm.conf
subPath: php-fpm.conf
- name: tdd-web-configmap
mountPath: /usr/local/etc/php-fpm.d/www.conf
subPath: www.conf
#挂载日志卷
- name: tdd-log
mountPath: /usr/local/var/log
volumes:
- name: tdd-web-configmap
configMap:
name: tdd-web-configmap
items:
- key: nginx.conf
path: nginx.conf
- key: php-fpm.conf
path: php-fpm.conf
- key: www.conf
path: www.conf
- name: tdd-web-data
persistentVolumeClaim:
claimName: tdd-web-pvc
- name: tdd-log
persistentVolumeClaim:
claimName: tdd-web-log-pvc
2.7 svc资源
apiVersion: v1
kind: Service
metadata:
name: tdd-web-svc
namespace: testphp
spec:
type: ClusterIP
selector:
app: tdd-web
ports:
- name: tdd-web-80
protocol: TCP
port: 80
targetPort: 80
- name: tdd-web-9000
protocol: TCP
port: 9000
targetPort: 9000
3.部署rabbitmq集群
namespace
kubectl create ns rabbitmq
3.1 ConfigMap
配置文件解释
#default_pass/default_pass：声明用户名和密码（虽然有部分文章记录可以通过环境变量的方式声明，但是经测试，针对此版本如果指定了configmap即rabbitmq的配置文件，声明的环境变量是没有用的，都需要在配置文件中指定）
#default_user = admin
#default_pass = admin123
#guest默认为只能本地登录,设置为false，表示可以远程登录
loopback_users.guest = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.rabbitmq.svc.cluster.local
#RabbitMQ节点名应该根据pod的主机名或IP地址进行设置，但是podIP地址不稳定，所以设置为hostname
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = rabbitmq-headless
#rabbitmq is rabbitmq-cluster’s namespace
cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local
#节点清理检测多长时间运行一次
cluster_formation.node_cleanup.interval = 30
#如果需要自动删除未知/不存在的节点，请设置为false。这可能很危险
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
#队列主节点的策略，有三大策略 min-masters，client-local，random
queue_master_locator = min-masters
#cluster_formation.randomized_startup_delay_range.min = 0
#cluster_formation.randomized_startup_delay_range.max = 2
#触发流量控制的内存阈值，可以为相对值(0.5),或者绝对值
vm_memory_high_watermark.absolute = 1GB
#Rabbitmq存储数据的可用空间限制，当低于该值的时候，将触发流量限制，设置可参考vm_memory_high_watermark参数
disk_free_limit.absolute = 2GB
#Source: rabbitmq-ha/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: rabbitmq
data:
enabled_plugins: |
[
rabbitmq_shovel,
rabbitmq_shovel_management,
rabbitmq_federation,
rabbitmq_federation_management,
rabbitmq_consistent_hash_exchange,
rabbitmq_management,
rabbitmq_peer_discovery_k8s
].
rabbitmq.conf: |
## RabbitMQ configuration
## Ref: https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.conf.example
## Authentification
## Clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.node_cleanup.interval = 10
# Set to false if automatic cleanup of absent nodes is desired.
# This can be dangerous, see http://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup.
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
## The default "guest" user is only permitted to access the server
## via a loopback interface (e.g. localhost)
loopback_users.guest = false
management.load_definitions = /etc/definitions/definitions.json
## Memory-based Flow Control threshold
vm_memory_high_watermark.absolute = 256MB
## Auth HTTP Backend Plugin
## LDAP Plugin
## MQTT Plugin
## Web MQTT Plugin
## STOMP Plugin
## Web STOMP Plugin
## Prometheus Plugin
## AMQPS support
3.2 Secret
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: "rabbitmq"
type: Opaque
data:
rabbitmq-username: "Z3Vlc3Q="
rabbitmq-password: "N09KV3JsSHpvdk5mZ3ZHQVNScTAyRnRX"
rabbitmq-management-username: "bWFuYWdlbWVudA=="
rabbitmq-management-password: "RUtEY1lCamdKR1d0VFBwR1pDa3lVZ2U3"
rabbitmq-erlang-cookie: "ZHlIdjJWbWVJdFB2S2JoRjhZeUEyRGFkODZzNnNFbkg="
definitions.json: "ewogICJnbG9iYWxfcGFyYW1ldGVycyI6IFsKICAgIAogIF0sCiAgInVzZXJzIjogWwogICAgewogICAgICAibmFtZSI6ICJtYW5hZ2VtZW50IiwKICAgICAgInBhc3N3b3JkIjogIkVLRGNZQmpnSkdXdFRQcEdaQ2t5VWdlNyIsCiAgICAgICJ0YWdzIjogIm1hbmFnZW1lbnQiCiAgICB9LAogICAgewogICAgICAibmFtZSI6ICJndWVzdCIsCiAgICAgICJwYXNzd29yZCI6ICI3T0pXcmxIem92TmZndkdBU1JxMDJGdFciLAogICAgICAidGFncyI6ICJhZG1pbmlzdHJhdG9yIgogICAgfQogIF0sCiAgInZob3N0cyI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiLyIKICAgIH0KICBdLAogICJwZXJtaXNzaW9ucyI6IFsKICAgIHsKICAgICAgInVzZXIiOiAiZ3Vlc3QiLAogICAgICAidmhvc3QiOiAiLyIsCiAgICAgICJjb25maWd1cmUiOiAiLioiLAogICAgICAicmVhZCI6ICIuKiIsCiAgICAgICJ3cml0ZSI6ICIuKiIKICAgIH0KICBdLAogICJwYXJhbWV0ZXJzIjogWwogICAgCiAgXSwKICAicG9saWNpZXMiOiBbCiAgICAKICBdLAogICJxdWV1ZXMiOiBbCiAgICAKICBdLAogICJleGNoYW5nZXMiOiBbCiAgICAKICBdLAogICJiaW5kaW5ncyI6IFsKICAgIAogIF0KfQ=="
- definitions.json
{
"global_parameters": [
],
"users": [
{
"name": "management",
"password": "EKDcYBjgJGWtTPpGZCkyUge7",
"tags": "management"
},
{
"name": "guest",
"password": "7OJWrlHzovNfgvGASRq02FtW",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "/"
}
],
"permissions": [
{
"user": "guest",
"vhost": "/",
"configure": ".*",
"read": ".*",
"write": ".*"
}
],
"parameters": [
],
"policies": [
],
"queues": [
],
"exchanges": [
],
"bindings": [
]
}
- rabbitmq-username
guest
- rabbitmq-password
7OJWrlHzovNfgvGASRq02FtW
- rabbitmq-management-password
EKDcYBjgJGWtTPpGZCkyUge7
- rabbitmq-management-username
management
- rabbitmq-erlang-cookie
dyHv2VmeItPvKbhF8YyA2Dad86s6sEnH
3.3 ServiceAccount
# Source: rabbitmq-ha/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: "rabbitmq"
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
automountServiceAccountToken: true
3.4 Role
# Source: rabbitmq-ha/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: "rabbitmq"
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get"]
3.5 RoleBinging
# Source: rabbitmq-ha/templates/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: "rabbitmq"
heritage: "Helm"
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
subjects:
- kind: ServiceAccount
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rabbitmq-rabbitmq-ha
Service
# Source: rabbitmq-ha/templates/service-discovery.yaml
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-rabbitmq-ha-discovery
namespace: rabbitmq
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: rabbitmq
spec:
clusterIP: None
ports:
- name: http
protocol: TCP
port: 15672
targetPort: http
- name: amqp
protocol: TCP
port: 5672
targetPort: amqp
- name: epmd
protocol: TCP
port: 4369
targetPort: epmd
publishNotReadyAddresses: true
selector:
app: rabbitmq-ha
release: rabbitmq
type: ClusterIP
3.6 Service
# Source: rabbitmq-ha/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: rabbitmq
heritage: Helm
spec:
ports:
- name: http
protocol: TCP
port: 15672
targetPort: http
- name: amqp
protocol: TCP
port: 5672
targetPort: amqp
- name: epmd
protocol: TCP
port: 4369
targetPort: epmd
selector:
app: rabbitmq-ha
release: rabbitmq
type: ClusterIP
3.7 StatefulSet
# Source: rabbitmq-ha/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: rabbitmq-rabbitmq-ha
namespace: rabbitmq
labels:
app: rabbitmq-ha
chart: rabbitmq-ha-1.36.4
release: rabbitmq
heritage: Helm
spec:
#pod管理策略，有状态的服务按顺序启动容器，例如master节点先启动
podManagementPolicy: OrderedReady
serviceName: rabbitmq-rabbitmq-ha-discovery
replicas: 3
updateStrategy:
type: OnDelete
selector:
matchLabels:
app: rabbitmq-ha
release: rabbitmq
template:
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
annotations:
checksum/config: 53a4503332f4f41b81943686ae339e73331cfee32b9216bbb2071ff1243650d5
checksum/secret: 742f34b4c06e7a69d056af6708b39515d44d84e36f85a2b6ccb6a9d85a36d371
spec:
terminationGracePeriodSeconds: 10
securityContext:
fsGroup: 101
runAsGroup: 101
runAsNonRoot: true
runAsUser: 100
serviceAccountName: rabbitmq-rabbitmq-ha
initContainers:
- name: bootstrap
image: busybox:1.30.1
imagePullPolicy: IfNotPresent
command: ['sh']
args:
- "-c"
- |
set -ex
cp /configmap/* /etc/rabbitmq
echo "${RABBITMQ_ERLANG_COOKIE}" > /var/lib/rabbitmq/.erlang.cookie
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: RABBITMQ_MNESIA_DIR
value: /var/lib/rabbitmq/mnesia/rabbit@$(POD_NAME).rabbitmq-rabbitmq-ha-discovery.rabbitmq（部署的命名空间的名字）.svc.cluster.local
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: rabbitmq-rabbitmq-ha
key: rabbitmq-erlang-cookie
resources:
{}
volumeMounts:
- name: configmap
mountPath: /configmap
- name: config
mountPath: /etc/rabbitmq
- name: data
mountPath: /var/lib/rabbitmq
containers:
- name: rabbitmq-ha
image: rabbitmq:3.8.0-alpine
imagePullPolicy: IfNotPresent
ports:
- name: epmd
protocol: TCP
containerPort: 4369
- name: amqp
protocol: TCP
containerPort: 5672
- name: http
protocol: TCP
containerPort: 15672
livenessProbe:
exec:
command:
- /bin/sh
- -c
- 'wget -O - -q --header "Authorization: Basic `echo -n \"$RABBIT_MANAGEMENT_USER:$RABBIT_MANAGEMENT_PASSWORD\"
| base64`" http://localhost:15672/api/healthchecks/node | grep -qF "{\"status\":\"ok\"}"'
failureThreshold: 6
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- /bin/sh
- -c
- 'wget -O - -q --header "Authorization: Basic `echo -n \"$RABBIT_MANAGEMENT_USER:$RABBIT_MANAGEMENT_PASSWORD\"
| base64`" http://localhost:15672/api/healthchecks/node | grep -qF "{\"status\":\"ok\"}"'
failureThreshold: 6
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 3
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: RABBITMQ_USE_LONGNAME
value: "true"
- name: RABBITMQ_NODENAME
value: rabbit@$(MY_POD_NAME).rabbitmq-rabbitmq-ha-discovery.rabbitmq（部署的命名空间）.svc.cluster.local
- name: K8S_HOSTNAME_SUFFIX
value: .rabbitmq-rabbitmq-ha-discovery.rabbitmq（部署的命名空间）.svc.cluster.local
- name: K8S_SERVICE_NAME
value: rabbitmq-rabbitmq-ha-discovery
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: rabbitmq-rabbitmq-ha
key: rabbitmq-erlang-cookie
- name: RABBIT_MANAGEMENT_USER
valueFrom:
secretKeyRef:
name: rabbitmq-rabbitmq-ha
key: rabbitmq-management-username
- name: RABBIT_MANAGEMENT_PASSWORD
valueFrom:
secretKeyRef:
name: rabbitmq-rabbitmq-ha
key: rabbitmq-management-password
resources:
{}
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq
- name: definitions
mountPath: /etc/definitions
readOnly: true
#requiredDuringSchedulingIgnoredDuringExecution：硬性要求，必须满足条件，保证分散部署的效果最好使用用此方式
# preferredDuringSchedulingIgnoredDuringExecution：软性要求，可以不完全满足，即有可能同一node上可以跑多个副本
affinity:
podAntiAffinity:
#如果节点上的pod标签存在满足app: rabbitmq-ha，release: rabbitmq，则不能部署到节点上
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: rabbitmq-ha
release: rabbitmq
volumes:
- name: config
emptyDir: {}
- name: configmap
configMap:
name: rabbitmq-rabbitmq-ha
- name: definitions
secret:
secretName: rabbitmq-rabbitmq-ha
items:
- key: definitions.json
path: definitions.json
volumeClaimTemplates:
- metadata:
name: data
annotations:
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"
storageClassName: "rabbitmq"
3.8 StorageClass
#如果使用本地磁盘这个就用不到
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rabbitmq
mountOptions:
- 'nolock,tcp,noresvport'
- vers=3
parameters:
path: /xxxxxx/rabbitmq
server: 666666666.cn-beijing.nas.aliyuncs.com
volumeAs: subpath
provisioner: nasplugin.csi.alibabacloud.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
3.9 PV
3.9.1 本地磁盘做PV
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-local-0
labels:
app: rabbitmq-data-0
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
#声明适合的pvc
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-0
namespace: rabbitmq
local:
path: /data/rabbitmq/data/data01
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node02
persistentVolumeReclaimPolicy: Retain
storageClassName: rabbitmq
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-local-1
labels:
app: rabbitmq-data-1
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-1
namespace: rabbitmq
local:
path: /data/rabbitmq/data/data02
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node02
persistentVolumeReclaimPolicy: Retain
storageClassName: rabbitmq
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-local-2
labels:
app: rabbitmq-data-2
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-2
namespace: rabbitmq
local:
path: /data/rabbitmq/data/data03
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node02
persistentVolumeReclaimPolicy: Retain
storageClassName: rabbitmq
3.9.2 StorageClass创建的PV
- data-rabbitmq-rabbitmq-ha-0
apiVersion: v1
kind: PersistentVolume
metadata:
name: nas-2f929e65-4585-4e1d-bbfe-ad5679526d46
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 8Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-0
namespace: pub
csi:
driver: nasplugin.csi.alibabacloud.com
fsType: ext4
volumeAttributes:
path: /nas-2f929e65-4585-4e1d-bbfe-ad5679526d46
server: 666666666.cn-beijing.nas.aliyuncs.com
storage.kubernetes.io/csiProvisionerIdentity: 1638948116227-8081-nasplugin.csi.alibabacloud.com
volumeAs: subpath
volumeHandle: nas-2f929e65-4585-4e1d-bbfe-ad5679526d46
mountOptions:
- 'nolock,tcp,noresvport'
- vers=3
persistentVolumeReclaimPolicy: Delete
storageClassName: rabbitmq
volumeMode: Filesystem
- data-rabbitmq-rabbitmq-ha-1
apiVersion: v1
kind: PersistentVolume
metadata:
name: nas-9ce64f6e-2377-49c5-925d-7c167ef11c4d
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 8Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-1
namespace: pub
csi:
driver: nasplugin.csi.alibabacloud.com
fsType: ext4
volumeAttributes:
path: /nas-9ce64f6e-2377-49c5-925d-7c167ef11c4d
server: 666666666.cn-beijing.nas.aliyuncs.com
storage.kubernetes.io/csiProvisionerIdentity: 1638948116227-8081-nasplugin.csi.alibabacloud.com
volumeAs: subpath
volumeHandle: nas-9ce64f6e-2377-49c5-925d-7c167ef11c4d
mountOptions:
- 'nolock,tcp,noresvport'
- vers=3
persistentVolumeReclaimPolicy: Delete
storageClassName: rabbitmq
volumeMode: Filesystem
- data-rabbitmq-rabbitmq-ha-2
apiVersion: v1
kind: PersistentVolume
metadata:
name: nas-03602164-e318-4ea0-93e3-aba48b2c9263
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 8Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: data-rabbitmq-rabbitmq-ha-2
namespace: pub
csi:
driver: nasplugin.csi.alibabacloud.com
fsType: ext4
volumeAttributes:
path: /nas-03602164-e318-4ea0-93e3-aba48b2c9263
server: 666666666.cn-beijing.nas.aliyuncs.com
storage.kubernetes.io/csiProvisionerIdentity: 1638948116227-8081-nasplugin.csi.alibabacloud.com
volumeAs: subpath
volumeHandle: nas-03602164-e318-4ea0-93e3-aba48b2c9263
mountOptions:
- 'nolock,tcp,noresvport'
- vers=3
persistentVolumeReclaimPolicy: Delete
storageClassName: rabbitmq
volumeMode: Filesystem
3.10 PVC
3.10.1 本地磁盘pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-0
namespace: rabbitmq
spec:
selector:
matchLabels:
app: rabbitmq-data-0
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: rabbitmq
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-1
namespace: rabbitmq
spec:
selector:
matchLabels:
app: rabbitmq-data-1
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: rabbitmq
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-2
namespace: rabbitmq
spec:
selector:
matchLabels:
app: rabbitmq-data-2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: rabbitmq
3.10.2 StorageClass中的pvc
- data-rabbitmq-rabbitmq-ha-0
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-0
namespace: rabbitmq
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: rabbitmq
volumeMode: Filesystem
volumeName: nas-c99ec219-c2af-41d7-9956-1cf83c60ffc8
- data-rabbitmq-rabbitmq-ha-1
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-1
namespace: rabbitmq
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: rabbitmq
volumeMode: Filesystem
volumeName: nas-8028a035-152c-40bf-a799-fd3f24211679
- data-rabbitmq-rabbitmq-ha-2
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: rabbitmq-ha
release: rabbitmq
name: data-rabbitmq-rabbitmq-ha-2
namespace: rabbitmq
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: rabbitmq
volumeMode: Filesystem
volumeName: nas-2afc1c88-8afa-4fd0-a9e5-74a0cffc6ab9
3.11 Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-weight: ''
generation: 3
name: rabbitmq
namespace: rabbitmq
spec:
rules:
- host: rabbitmq.timerovers.com
http:
paths:
- backend:
service:
name: rabbit-svc
port:
number: 15672
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- rabbitmq.timerovers.com
secretName: timerovers.tls
status:
loadBalancer:
ingress:
- ip: 213.13.33.3
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献6条内容
所有评论(0)