前言

本文旨在记录使用K8s部署Rocketmq中各种坑和注意事项,也参考了不是同站其他博主的帖子,但均不是特别满足我的需求,因此对他们的方案进行的一定的修改。本文,以一个虚拟的财务金融项目部署mq需求为例。

准备工作,拉取镜像

	选取的镜像版本为:
  apache/rocketmq:5.2.0
  rocketmq-dashboard:1.0.0

1、基于原有镜像编译新镜像

主要是为了方便快速扩展而 使用,非必须,但是在本文是很重要的前置步骤。

运行一个镜像拷贝出runbroker.sh、runserver.sh

注:仅展示修改项,折叠的为原始文件内容。

runbroker.sh 修改如下

#!/bin/bash


#复制挂载文件
cp /path/to/mounted/config/broker.conf /home/rocketmq/broker.conf

ORDINAL=${POD_NAME##*-}
# 替换brokerID
sed -i "s/\${BROKER_ID}/${ORDINAL}/g" /home/rocketmq/broker.conf
# 替换配置文件中的 ${POD_NAME}
sed -i "s/\${POD_NAME}/${POD_NAME}/g" /home/rocketmq/broker.conf

#注释动态计算堆。改成从jvm参数指明
#calculate_heap_sizes

runserver.sh修改如下:

#!/bin/bash
#注释动态计算堆。改成从jvm参数指明
#calculate_heap_sizes

Dockfile


FROM apache/rocketmq:5.2.0

WORKDIR /home/rocketmq/rocketmq-5.2.0/bin

#启动broker脚本
COPY runbroker.sh /home/rocketmq/rocketmq-5.2.0/bin

#启动nameserver脚本
COPY runserver.sh /home/rocketmq/rocketmq-5.2.0/bin

构建镜像并推送至自建的Harbor。

2、创建k8s SC(StorageClass)存储类

我这里采用openebs的本地方式实现存储类。

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-rocketmq
  annotations:
    openebs.io/cas-type: local
    storageclass.kubernetes.io/is-default-class: "false"
    cas.openebs.io/config: |
      #hostpath type will create a PV by
      # creating a sub-directory under the
      # BASEPATH provided below.
      - name: StorageType
        value: "hostpath"
      #Specify the location (directory) where
      # where PV(volume) data will be saved.
      # A sub-directory with pv-name will be
      # created. When the volume is deleted,
      # the PV sub-directory will be deleted.
      #Default value is /var/openebs/local
      - name: BasePath
        value: "/data/openebs/rocketmq"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

查看验证SC

kubectl get sc|grep mq
openebs-rocketmq            openebs.io/local   Retain          WaitForFirstConsumer   false                  17h

3、部署nameserver

先创建命名空间

#创建namespace
kubectl create ns finance-envir

nameserver的部署比较简单,直接放yaml

注: 创建headless Service,专用于内部节点寻址,不对外暴露。因最终mq客户端也在k8s集群,因此nameserver和broker均不对暴露,仅dashboard对外提供nodeport方式访问方式,便于观测维护集群。

kubectl apply -f 执行文件

headless yaml:

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-name-src-service
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-name-srv
  ports:
    - name: name-service-port
      port: 9876
      protocol: TCP
  clusterIP: None

namesrv yaml:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-name-srv
  namespace: finance-envir
  labels:
    app: finance-rocketmq-name-srv
spec:
  serviceName: 'rocketmq-name-src-service'
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-name-srv
  template:
    metadata:
      labels:
        app: finance-rocketmq-name-srv
    spec:
      containers:
        - name: finance-rocketmq-name-srv
          image: 'apache/rocketmq:5.2.0'
          command: ["sh", "mqnamesrv"]
          ports:
            - name: tcp-9876
              containerPort: 9876
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 50m
              memory: 256Mi
          volumeMounts:
            - name: rocketmq-namesrv-storage
              mountPath: /home/rocketmq/logs
              subPath: logs
          imagePullPolicy: Always
  volumeClaimTemplates:
    - metadata:
        name: rocketmq-namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: openebs-rocketmq
        resources:
          requests:
            storage: 20Gi

4、部署master

  1. 创建configmap

yaml如下:

kind: ConfigMap
apiVersion: v1
metadata:
  name: finance-rocketmq-broker-master-config
  namespace: finance-envir
data:
  broker.conf: |-
    brokerClusterName = DefaultCluster
    brokerId = 0
    brokerName = finance-rocketmq-${BROKER_ID}
    brokerIp1 = ${POD_NAME}.finance-rocketmq-broker-master.finance-envir.svc.cluster.local
    deleteWhen = 04
    fileReservedTime = 48
    brokerRole = ASYNC_MASTER
    flushDiskType = ASYNC_FLUSH

上述在配置文件中使用了占位符,因为Rocketmq是通过brokerName 来联系主从,为了方便快捷扩展一主一从模式,因此采用了占位符。后面有演示效果。而Ip则设置为通过headless可以查询的方式,即通过podname.service.namsespace.xxx的方式,在部署statefulSet中常用。

  1. 创建headless service
    yaml如下:
apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-broker-master
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-broker-master
  ports:
    - name: tcp-vip-10909
      port: 10909
      protocol: TCP
    - name: tcp-vip-10911
      port: 10911
      protocol: TCP
    - name: tcp-vip-10912
      port: 10912
      protocol: TCP
  clusterIP: None
  1. 创建 broker
    yaml:
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-broker-master
  namespace: finance-envir
  labels:
    app: finance-rocketmq-broker-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-broker-master
  template:
    metadata:
      labels:
        app: finance-rocketmq-broker-master
    spec:
      volumes:
        - name: rocketmq-broker-config
          configMap:
            name: finance-rocketmq-broker-master-config
            items:
              - key: broker.conf
                path: broker.conf
          #  defaultMode: 420
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rocketmq-broker
          image: '你编译后的镜像地址'
          command: ["sh","mqbroker","-c","/home/rocketmq/broker.conf"]
          ports:
            - name: tcp-vip-10909
              containerPort: 10909
              protocol: TCP
            - name: tcp-main-10911
              containerPort: 10911
              protocol: TCP
            - name: tcp-ha-10912
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR
              value: finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876
            - name: MAX_HEAP_SIZE
              value: 512M
            - name: HEAP_NEWSIZE
              value: 256M
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          resources:
            limits:
              cpu: 500m
              memory: 800Mi
            requests:
              cpu: 250m
              memory: 800Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/logs
              subPath: logs/broker-0-master
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/store
              subPath: store/broker-0-master
            - name: rocketmq-broker-config
              mountPath: /path/to/mounted/config/broker.conf
              subPath: broker.conf
          imagePullPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: rocketmq-broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-rocketmq
        volumeMode: Filesystem
  serviceName: 'finance-rocketmq-broker-master'

注:

  • broker中使用Downward API将当前的podname注入进去,从而结合我们自定义的脚本更改占位符
  • 通过volumeClaimTemplates声明使用存储类以及限制大小
  • 指明nameserver地址,集群nameserver模式可以指明多个
  • 镜像为带我们修改后脚本新编译的镜像

5、部署slave

configmap yaml:

kind: ConfigMap
apiVersion: v1
metadata:
  name: finance-rocketmq-broker-slave-config
  namespace: finance-envir
data:
  broker.conf: |-
    brokerClusterName = DefaultCluster
    brokerId = 1
    brokerName = finance-rocketmq-${BROKER_ID}
    brokerIp1 = ${POD_NAME}.finance-rocketmq-broker-slave.finance-envir.svc.cluster.local
    deleteWhen = 04
    fileReservedTime = 48
    brokerRole = SLAVE
    flushDiskType = ASYNC_FLUSH

headless service yaml:

apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-broker-slave
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-broker-slave
  ports:
    - name: tcp-vip-10909
      port: 10909
      protocol: TCP
    - name: tcp-vip-10911
      port: 10911
      protocol: TCP
    - name: tcp-vip-10912
      port: 10912
      protocol: TCP
  clusterIP: None

broker yaml:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-broker-slave
  namespace: finance-envir
  labels:
    app: finance-rocketmq-broker-slave
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-broker-slave
  template:
    metadata:
      labels:
        app: finance-rocketmq-broker-slave
    spec:
      volumes:
        - name: rocketmq-broker-config
          configMap:
            name: finance-rocketmq-broker-slave-config
            items:
              - key: broker.conf
                path: broker.conf
          #  defaultMode: 420
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rocketmq-broker
          image: '修改后的镜像'
          command: ["sh","mqbroker","-c","/home/rocketmq/broker.conf"]
          ports:
            - name: tcp-vip-10909
              containerPort: 10909
              protocol: TCP
            - name: tcp-main-10911
              containerPort: 10911
              protocol: TCP
            - name: tcp-ha-10912
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR
              value: finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876
            - name: MAX_HEAP_SIZE
              value: 512M
            - name: HEAP_NEWSIZE
              value: 256M
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          resources:
            limits:
              cpu: 500m
              memory: 800Mi
            requests:
              cpu: 250m
              memory: 800Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/logs
              subPath: logs/broker-0-master
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/store
              subPath: store/broker-0-master
            - name: rocketmq-broker-config
              mountPath: /path/to/mounted/config/broker.conf
              subPath: broker.conf
          imagePullPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: rocketmq-broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-rocketmq
        volumeMode: Filesystem
  serviceName: 'finance-rocketmq-broker-slave'

对比master,仅在配置文件处指明了slave的相关信息,仅brokerId 为1的可以支持读写,并设置角色为slave。

6、部署dashboard

检查上诉各实例不出错的情况下,可以部署dashboard进行可视化观察。
yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: finance-rocketmq-dashboard
  namespace: finance-envir
  labels:
    app: finance-rocketmq-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-dashboard
  template:
    metadata:
      name: finance-rocketmq-dashboard
      labels:
        app: finance-rocketmq-dashboard
    spec:
      containers:
        - name: finance-rocketmq-dashboard
          image: apacherocketmq/rocketmq-dashboard:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: dashboard-web
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: ' -Xmx256M -Xms256M -Xmn128M -Drocketmq.namesrv.addr=finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false'
      restartPolicy: Always


---

apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-dashboard
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-dashboard
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 32080
  type: NodePort

7、可视化观测
如果一切成功,即可访问 你的k8s节点的32080端口

标准集群一主一从:

在这里插入图片描述
通过yaml配置文件或者kubesphere等k8s管理工具,分别将master和slave增加一个副本,即可变为2主2从部署结构(因为自定义脚本以及statefulset的pod_name后缀是数字递增的原因,通过替换即可完成对应关系)。

在这里插入图片描述

从topic观测集群:

一主一从主题:
在这里插入图片描述
两主两从主题:
在这里插入图片描述

可见,可以通过修改statefulSet的实例数,迅速增加新的主从设置,不需要重新部署。

后言:

  1. 如果需要实现一个主多个从,可以采用类似的占位符方式,固定一部分,读取statefulSet实例的数字段+1方式,实现一主多从。但是brokerName需要固定。其他变种参考类似实现方式。
  2. 本部署方式尚未经过大规模项目测试,实际使用需要结合自己的情况,本文仅做交流记录、分享。
Logo

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐