K8S部署可以快速扩展Rocketmq集群

前言

本文旨在记录使用K8s部署Rocketmq中各种坑和注意事项，也参考了不是同站其他博主的帖子，但均不是特别满足我的需求，因此对他们的方案进行的一定的修改。本文，以一个虚拟的财务金融项目部署mq需求为例。

准备工作，拉取镜像

	选取的镜像版本为：
  apache/rocketmq:5.2.0
  rocketmq-dashboard:1.0.0

1、基于原有镜像编译新镜像

主要是为了方便快速扩展而 使用，非必须，但是在本文是很重要的前置步骤。

运行一个镜像拷贝出runbroker.sh、runserver.sh

注：仅展示修改项，折叠的为原始文件内容。

runbroker.sh 修改如下

#!/bin/bash


#复制挂载文件
cp /path/to/mounted/config/broker.conf /home/rocketmq/broker.conf

ORDINAL=${POD_NAME##*-}
# 替换brokerID
sed -i "s/\${BROKER_ID}/${ORDINAL}/g" /home/rocketmq/broker.conf
# 替换配置文件中的 ${POD_NAME}
sed -i "s/\${POD_NAME}/${POD_NAME}/g" /home/rocketmq/broker.conf

#注释动态计算堆。改成从jvm参数指明
#calculate_heap_sizes

runserver.sh修改如下：

#!/bin/bash
#注释动态计算堆。改成从jvm参数指明
#calculate_heap_sizes

Dockfile


FROM apache/rocketmq:5.2.0

WORKDIR /home/rocketmq/rocketmq-5.2.0/bin

#启动broker脚本
COPY runbroker.sh /home/rocketmq/rocketmq-5.2.0/bin

#启动nameserver脚本
COPY runserver.sh /home/rocketmq/rocketmq-5.2.0/bin

构建镜像并推送至自建的Harbor。

2、创建k8s SC（StorageClass）存储类

我这里采用openebs的本地方式实现存储类。

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-rocketmq
  annotations:
    openebs.io/cas-type: local
    storageclass.kubernetes.io/is-default-class: "false"
    cas.openebs.io/config: |
      #hostpath type will create a PV by
      # creating a sub-directory under the
      # BASEPATH provided below.
      - name: StorageType
        value: "hostpath"
      #Specify the location (directory) where
      # where PV(volume) data will be saved.
      # A sub-directory with pv-name will be
      # created. When the volume is deleted,
      # the PV sub-directory will be deleted.
      #Default value is /var/openebs/local
      - name: BasePath
        value: "/data/openebs/rocketmq"
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

查看验证SC

kubectl get sc|grep mq
openebs-rocketmq            openebs.io/local   Retain          WaitForFirstConsumer   false                  17h

3、部署nameserver

先创建命名空间

#创建namespace
kubectl create ns finance-envir

nameserver的部署比较简单，直接放yaml

注： 创建headless Service，专用于内部节点寻址，不对外暴露。因最终mq客户端也在k8s集群，因此nameserver和broker均不对暴露，仅dashboard对外提供nodeport方式访问方式,便于观测维护集群。

kubectl apply -f 执行文件

headless yaml:

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-name-src-service
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-name-srv
  ports:
    - name: name-service-port
      port: 9876
      protocol: TCP
  clusterIP: None

namesrv yaml:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-name-srv
  namespace: finance-envir
  labels:
    app: finance-rocketmq-name-srv
spec:
  serviceName: 'rocketmq-name-src-service'
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-name-srv
  template:
    metadata:
      labels:
        app: finance-rocketmq-name-srv
    spec:
      containers:
        - name: finance-rocketmq-name-srv
          image: 'apache/rocketmq:5.2.0'
          command: ["sh", "mqnamesrv"]
          ports:
            - name: tcp-9876
              containerPort: 9876
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 50m
              memory: 256Mi
          volumeMounts:
            - name: rocketmq-namesrv-storage
              mountPath: /home/rocketmq/logs
              subPath: logs
          imagePullPolicy: Always
  volumeClaimTemplates:
    - metadata:
        name: rocketmq-namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: openebs-rocketmq
        resources:
          requests:
            storage: 20Gi

4、部署master

  1. 创建configmap

yaml如下：

kind: ConfigMap
apiVersion: v1
metadata:
  name: finance-rocketmq-broker-master-config
  namespace: finance-envir
data:
  broker.conf: |-
    brokerClusterName = DefaultCluster
    brokerId = 0
    brokerName = finance-rocketmq-${BROKER_ID}
    brokerIp1 = ${POD_NAME}.finance-rocketmq-broker-master.finance-envir.svc.cluster.local
    deleteWhen = 04
    fileReservedTime = 48
    brokerRole = ASYNC_MASTER
    flushDiskType = ASYNC_FLUSH

上述在配置文件中使用了占位符，因为Rocketmq是通过brokerName 来联系主从，为了方便快捷扩展一主一从模式，因此采用了占位符。后面有演示效果。而Ip则设置为通过headless可以查询的方式，即通过podname.service.namsespace.xxx的方式，在部署statefulSet中常用。

  1. 创建headless service
    yaml如下：
apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-broker-master
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-broker-master
  ports:
    - name: tcp-vip-10909
      port: 10909
      protocol: TCP
    - name: tcp-vip-10911
      port: 10911
      protocol: TCP
    - name: tcp-vip-10912
      port: 10912
      protocol: TCP
  clusterIP: None
  1. 创建 broker
    yaml：
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-broker-master
  namespace: finance-envir
  labels:
    app: finance-rocketmq-broker-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-broker-master
  template:
    metadata:
      labels:
        app: finance-rocketmq-broker-master
    spec:
      volumes:
        - name: rocketmq-broker-config
          configMap:
            name: finance-rocketmq-broker-master-config
            items:
              - key: broker.conf
                path: broker.conf
          #  defaultMode: 420
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rocketmq-broker
          image: '你编译后的镜像地址'
          command: ["sh","mqbroker","-c","/home/rocketmq/broker.conf"]
          ports:
            - name: tcp-vip-10909
              containerPort: 10909
              protocol: TCP
            - name: tcp-main-10911
              containerPort: 10911
              protocol: TCP
            - name: tcp-ha-10912
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR
              value: finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876
            - name: MAX_HEAP_SIZE
              value: 512M
            - name: HEAP_NEWSIZE
              value: 256M
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          resources:
            limits:
              cpu: 500m
              memory: 800Mi
            requests:
              cpu: 250m
              memory: 800Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/logs
              subPath: logs/broker-0-master
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/store
              subPath: store/broker-0-master
            - name: rocketmq-broker-config
              mountPath: /path/to/mounted/config/broker.conf
              subPath: broker.conf
          imagePullPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: rocketmq-broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-rocketmq
        volumeMode: Filesystem
  serviceName: 'finance-rocketmq-broker-master'

注：

  • broker中使用Downward API将当前的podname注入进去，从而结合我们自定义的脚本更改占位符
  • 通过volumeClaimTemplates声明使用存储类以及限制大小
  • 指明nameserver地址，集群nameserver模式可以指明多个
  • 镜像为带我们修改后脚本新编译的镜像

5、部署slave

configmap yaml:

kind: ConfigMap
apiVersion: v1
metadata:
  name: finance-rocketmq-broker-slave-config
  namespace: finance-envir
data:
  broker.conf: |-
    brokerClusterName = DefaultCluster
    brokerId = 1
    brokerName = finance-rocketmq-${BROKER_ID}
    brokerIp1 = ${POD_NAME}.finance-rocketmq-broker-slave.finance-envir.svc.cluster.local
    deleteWhen = 04
    fileReservedTime = 48
    brokerRole = SLAVE
    flushDiskType = ASYNC_FLUSH

headless service yaml:

apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-broker-slave
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-broker-slave
  ports:
    - name: tcp-vip-10909
      port: 10909
      protocol: TCP
    - name: tcp-vip-10911
      port: 10911
      protocol: TCP
    - name: tcp-vip-10912
      port: 10912
      protocol: TCP
  clusterIP: None

broker yaml:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: finance-rocketmq-broker-slave
  namespace: finance-envir
  labels:
    app: finance-rocketmq-broker-slave
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-broker-slave
  template:
    metadata:
      labels:
        app: finance-rocketmq-broker-slave
    spec:
      volumes:
        - name: rocketmq-broker-config
          configMap:
            name: finance-rocketmq-broker-slave-config
            items:
              - key: broker.conf
                path: broker.conf
          #  defaultMode: 420
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rocketmq-broker
          image: '修改后的镜像'
          command: ["sh","mqbroker","-c","/home/rocketmq/broker.conf"]
          ports:
            - name: tcp-vip-10909
              containerPort: 10909
              protocol: TCP
            - name: tcp-main-10911
              containerPort: 10911
              protocol: TCP
            - name: tcp-ha-10912
              containerPort: 10912
              protocol: TCP
          env:
            - name: NAMESRV_ADDR
              value: finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876
            - name: MAX_HEAP_SIZE
              value: 512M
            - name: HEAP_NEWSIZE
              value: 256M
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          resources:
            limits:
              cpu: 500m
              memory: 800Mi
            requests:
              cpu: 250m
              memory: 800Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/logs
              subPath: logs/broker-0-master
            - name: rocketmq-broker-storage
              mountPath: /home/rocketmq/store
              subPath: store/broker-0-master
            - name: rocketmq-broker-config
              mountPath: /path/to/mounted/config/broker.conf
              subPath: broker.conf
          imagePullPolicy: Always
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: rocketmq-broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-rocketmq
        volumeMode: Filesystem
  serviceName: 'finance-rocketmq-broker-slave'

对比master，仅在配置文件处指明了slave的相关信息，仅brokerId 为1的可以支持读写，并设置角色为slave。

6、部署dashboard

检查上诉各实例不出错的情况下，可以部署dashboard进行可视化观察。
yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: finance-rocketmq-dashboard
  namespace: finance-envir
  labels:
    app: finance-rocketmq-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: finance-rocketmq-dashboard
  template:
    metadata:
      name: finance-rocketmq-dashboard
      labels:
        app: finance-rocketmq-dashboard
    spec:
      containers:
        - name: finance-rocketmq-dashboard
          image: apacherocketmq/rocketmq-dashboard:1.0.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
              name: dashboard-web
              protocol: TCP
          env:
            - name: JAVA_OPTS
              value: ' -Xmx256M -Xms256M -Xmn128M -Drocketmq.namesrv.addr=finance-rocketmq-name-srv-0.rocketmq-name-src-service.finance-envir.svc.cluster.local:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false'
      restartPolicy: Always


---

apiVersion: v1
kind: Service
metadata:
  name: finance-rocketmq-dashboard
  namespace: finance-envir
spec:
  selector:
    app: finance-rocketmq-dashboard
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 32080
  type: NodePort

7、可视化观测
如果一切成功，即可访问 你的k8s节点的32080端口

标准集群一主一从：

在这里插入图片描述
通过yaml配置文件或者kubesphere等k8s管理工具，分别将master和slave增加一个副本，即可变为2主2从部署结构（因为自定义脚本以及statefulset的pod_name后缀是数字递增的原因，通过替换即可完成对应关系）。

在这里插入图片描述

从topic观测集群：

一主一从主题：
在这里插入图片描述
两主两从主题：
在这里插入图片描述

可见，可以通过修改statefulSet的实例数，迅速增加新的主从设置，不需要重新部署。

后言：

  1. 如果需要实现一个主多个从，可以采用类似的占位符方式，固定一部分，读取statefulSet实例的数字段+1方式，实现一主多从。但是brokerName需要固定。其他变种参考类似实现方式。
  2. 本部署方式尚未经过大规模项目测试，实际使用需要结合自己的情况，本文仅做交流记录、分享。
# kubernetes # rocketmq # 容器
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes