基于k8s operator的mongodb多副本容器化部署实践

mongodb在实际应用中，经常用到。为了提供高可用的服务，经常会采用多副本的方式来部署实例。k8s这些年作为云原生服务的基石，深刻改变着运维与开发的使用场景。本文结合
mongodb-kubernetes-operator,基于k8s CRD(扩展资源) 来实现 mongodb多副本部署的开发运维。本文是基于mongodb社区提供的MongoDB Community Kubernetes Operator来实现相应的操作的，重要的是解决了官方提供的操作步骤中的一些坑，减少大家的试错时间。废话不多说进入正文.

1.创建命名空间

kubectl create ns mongodb

2.下载代码 (本文以0.7.0为例子)

git clone -b v0.7.0 https://github.com/mongodb/mongodb-kubernetes-operator.git

3.在mongodb名空间下，创建role、role binding 、service account

cd /root/dly/mongodb-kubernetes-operator
kubectl apply -k config/rbac --namespace mongodb

4.生成CRD(Custom Resource Definitions)

kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

5.确认第3步、第4步是否成功

#crd是否创建成功
kubectl get crd/mongodbcommunity.mongodbcommunity.mongodb.com
#角色是否成功
kubectl get role mongodb-kubernetes-operator --namespace mongodb
#角色绑定是否成功
kubectl get rolebinding mongodb-kubernetes-operator --namespace mongodb
#service账户是否成功
kubectl get serviceaccount mongodb-kubernetes-operator --namespace mongodb

6.安装operator

kubectl create -f config/manager/manager.yaml --namespace mongodb
#确认是否成功
kubectl get pods --namespace mongodb

7.部署mongodb多副本实例

到了重点的部分，根据git上的readme部署出现了一些列问题，坑不少，所以这个地方，直接贴出来相应的yaml文件，避免大家踩坑。

1. 创建pv,其中的大小暂定为1GB，大家可以安装自己的情况修改，本处仅为演示.

因为我们打算起来3个实例，每个实例用到了两个Volume,所以创建了6个PV.内容如下.这块git上是没有的。当然你也可以选择不用manual,可以换成其他的storageClass,如果他们有provisoner,你可以不事先创建pv.

然后执行命令kubectl apply -f pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
name: data-volume-example-mongodb-0
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-0"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: data-volume-example-mongodb-1
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-1"

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: data-volume-example-mongodb-2
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-2"


---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-volume-example-mongodb-0
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-agent-0"

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-volume-example-mongodb-1
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-agent-1"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: logs-volume-example-mongodb-2
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/dly/mongo-agent-2"

2.修改 config/samples下面的mongodb.com_v1_mongodbcommunity_cr.yaml，内容如下

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: example-mongodb
spec:
members: 3
type: ReplicaSet
version: "4.2.6"
security:
authentication:
modes: ["SCRAM"]
users:
- name: my-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-user-password
roles:
- name: clusterAdmin
db: admin
- name: userAdminAnyDatabase
db: admin
scramCredentialsSecretName: my-scram
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
statefulSet:
spec:
template:
spec:
containers:
- name: mongod
resources:
limits:
cpu: "0.2"
memory: 250M
requests:
cpu: "0.2"
memory: 200M
securityContext:
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
fsGroup: 0
- name: mongodb-agent
readinessProbe:
timeoutSeconds: 3
resources:
limits:
cpu: "0.2"
memory: 250M
requests:
cpu: "0.2"
memory: 200M
securityContext:
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
fsGroup: 0
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
storageClassName: manual
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
- metadata:
name: logs-volume
spec:
storageClassName: manual
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
#the user credentials will be generated from this secret
#once the credentials are generated, this secret is no longer required
---
apiVersion: v1
kind: Secret
metadata:
name: my-user-password
type: Opaque
stringData:
password: Cgyun_mongo

执行命令：

kubectl apply -f mongodb.com_v1_mongodbcommunity_cr.yaml -n mongodb

然后执行下述命令，发现三个pod起来了。

kubectl get pods -n mongodb

第4个pod是我们的operator. 同时我们看到在mongodb名空间下，svc也创建成功了，客户端可以通过该svc连接背后的pod

8.连接测试

在mongodb.com_v1_mongodbcommunity_cr.yaml文件中，可以看到如下内容。这两部分内容决定了连接客户端的用户名+密码. 其中用户名是my-user,密码是My_mongo。利用下面的信息便可以连接mongodb。

users:
- name: my-user
db: admin
passwordSecretRef: # a reference to the secret that will be used to generate the user's password
name: my-user-password
---
apiVersion: v1
kind: Secret
metadata:
name: my-user-password
type: Opaque
stringData:
password: Cgyun_mongo

运行下面的命令，进入其中一个pod,然后便可以连接数据库。

kubectl -n mongodb exec --stdin --tty example-mongodb-0 -- /bin/bash
mongo -u my-user -p Cgyun_mongo

9.参考资料

https://github.com/mongodb/mongodb-kubernetes-operator