Kubernetes部署大数据组件系列一：一键部署Zookeeper集群
目前的大数据平台存在的问题：通过手工方式直接部署在物理机上，过程复杂、效率低，不适合大规模集群的部署和动态扩容。由于平台组件繁多，对宿主机底层库依赖过高，一旦安装完成，很难进行迁移，也无法做成模版进行重复使用。组件升级不够灵活，往往会对有相同底层依赖的其他组件带来影响。采用虚机部署时，会对宿主机产生资源浪费、开销大。物理机或虚机关启等管理操作速度较慢。无法在同一套物理集群上部署多套大
·
目前的大数据平台存在的问题：
- 通过手工方式直接部署在物理机上，过程复杂、效率低，不适合大规模集群的部署和动态扩容。
- 由于平台组件繁多，对宿主机底层库依赖过高，一旦安装完成，很难进行迁移，也无法做成模版进行重复使用。
- 组件升级不够灵活，往往会对有相同底层依赖的其他组件带来影响。
- 采用虚机部署时，会对宿主机产生资源浪费、开销大。
- 物理机或虚机关启等管理操作速度较慢。
- 无法在同一套物理集群上部署多套大数据平台，使用虚机的方案太耗费资源。
针对上述问题，我们打算用时下流行的Docker容器来封装大数据各个组件，用Google的神器Kubernetes来编排这些Docker容器，最终达到：
- 平台能够自动化的快速部署（小规模集群能够达到秒级）、动态扩容。
- 平台构建一次，就能够形成模版重复使用、灵活迁移。
- 平台构建一次，就能够灵活部署在不同的环境中（比如公有云、私有云、裸机，虚拟机）。
- 组件能够灵活升级、灵活替代。
- 消除线上线下环境不一致导致的问题，再也不会有“在我的机器上运行的好好的”这种托词。
- 因为其多资源的低消耗，在物理集群上能同时部署多套大数据平台，供不同部门独 享。
- 很适合持续集成/持续部署(CI/CD)的环境，因为可以自动的下载、绿色的安装和彻底清除应用程序，不会影响宿主机的环境。
本篇是该系列的第一篇，手把手教你通过K8s部署zookeeper集群，涉及到的概念，各位看官自行脑补，我只讲一下部署过程，直接上干货了。
- 使用Kargo搭建Kubernetes 1.6集群:
![]()
地址 角色 10.0.7.14 Ansible Client （Kargo用到的工具） 10.0.8.182 Master, NODE 1 10.0.8.183 NODE 2 [下文提到的kargo.tg、kargo_images_v1.6.1.tar.gz压缩包]http://pan.baidu.com/s/1kUKyfBL
1，我用笔记本（macOS系统），作为Ansible client。先新建一个文件夹kargo，把kargo.tgz放进去并解压，目录结构如下：
![]()
2，需要安装python 2.7、pip 9.0.1、python-netaddr、ansible 2.3.1.0等工具 [ansible安装]http://valdhaus.co/writings/ansible-mac-osx/
3，修改 inventory/inventory.cfg
wangliangdeMacBook-Pro:kargo wangliang$ pwd /Users/wangliang/kargo/kargo wangliangdeMacBook-Pro:kargo wangliang$ vim inventory/inventory.cfg [all] node1 ansible_user=app ansible_host=10.0.8.182 ip=10.0.8.182 access_ip=10.0.8.182 node2 ansible_user=app ansible_host=10.0.8.183 ip=10.0.8.183 access_ip=10.0.8.183 [kube-master] node1 ip=10.0.8.182 access_ip=10.0.8.182 [kube-node] node1 node2 [etcd] node1 [k8s-cluster:children] kube-node kube-master [calico-rr]4，分别在10.0.8.182(Node 1)、10.0.8.183(Node 2)上做如下操作：
首先安装Docker 1.13.1，并启动服务
[Ubuntu安装Docker]http://www.linuxidc.com/Linux/2017-01/139985.htm其次把kargo_images_v1.6.1.tar.gz 拷贝到要安装k8s的节点上，放到k8s文件夹内，并解压，目录结构如下：
![]()
![]()
5，在kargo_images_v1.6.1目录下，进入bash，执行：➜ k8s cd kargo_images_v1.6.1 ➜ kargo_images_v1.6.1 pwd /data/k8s/kargo_images_v1.6.1 ➜ kargo_images_v1.6.1 bash app@yuyan1:/data/k8s/kargo_images_v1.6.1$images=$(ls -l ../kargo_images_v1.6.1|awk -F' ' '{ print $9 }') app@yuyan1:/data/k8s/kargo_images_v1.6.1$ for x in ${images[*]}; do sudo docker load -i $x; done将目录下的镜像放入docker images。
![]()
6， Node1、Node2和Mac间都要SSH免密，Mac上还要打开共享里面的允许远端访问
![]()
Node1、Node2、Mac都在用户目录下执行➜ ~ cd /home/app ssh-keygen -t rsa #一路回车就行 Generating public/private rsa key pair. Enter file in which to save the key (/home/app/.ssh/id_rsa): Created directory '/home/app/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/app/.ssh/id_rsa. Your public key has been saved in /home/app/.ssh/id_rsa.pub. The key fingerprint is: 7c:f4:2f:21:f2:85:7f:20:38:02:1d:ef:79:39:b6:be root@dcos1 The key's randomart image is: +--[ RSA 2048]----+ | . | | . o | | . . . . | | . o + + | | . S X = | | . B B + | | o o o | | . o | | E. | +-----------------+ ➜ ~ cd /home/app/.ssh ➜ .ssh ssh-copy-id Node2 ➜ .ssh ssh-copy-id Mac 每次执行，步骤都如下类似： The authenticity of host 'Node1 (10.0.8.183)' can't be established. ECDSA key fingerprint is 91:1a:13:a8:57:2b:a0:42:4d:aa:c9:83:c3:33:16:f9. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@dcos1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'Node2'" and check to make sure that only the key(s) you wanted were added.所有虚机都执行完成后，可以通过ssh hostname来查看之间是否ssh连接已经免密了，如下测试
➜ ~ ssh 10.0.8.183 Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-62-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage 180 packages can be updated. 75 updates are security updates. Last login: Wed Jun 28 09:16:54 2017 from 10.0.7.14 ➜ ~7，在Ansible client(Mac)上执行
$ ansible-playbook -i ~/kargo/inventory/inventory.cfg cluster.yml -b -v --user=username --ask-sudo-pass安装成功，在Mac上可以看到如下输出：
![]()
在Node 1上可以看到：
![]()
遇到的错误：
1，有如下错误：kubectl The connection to the server localhost:8080 was refused修改了vim inventory/group_vars/k8s-cluster.ym 里面的8080为28080，避免和别人冲突，又有如下错误：
kubernetes/master : Master | wait for the apiserver to be running原因是apiserver的镜像起不了，无法提供服务
怀疑可能是不支持docker 1.17版本，所以更换为docker 1.13.1版本子再次安装后没有上述错误了，安装成功。切记，要用docker 1.13.1版本。2，如果按照上述操作始终没有成功，可以把节点上的
/run/kubernetes /etc/kubernetes都删除，然后把docker也都卸载重新装。然后在Ansible client节点上重新部署就可以啦。
- 使用Dockerfile制作包含zkui 2.0 的Zookeeper 3.4.10版本的镜像：
1，建立你要做docker的目录，结构如下：
![]()
2，其中Dockerfile如下：
FROM openjdk:8-jre-alpine MAINTAINER Wang Liang <wangl8@knownsec.com> # Install required packages RUN apk add --no-cache \ bash \ su-exec ENV ZOO_USER=zookeeper \ ZOO_CONF_DIR=/conf \ ZOO_DATA_DIR=/data \ ZOO_UI_DIR=/zkui \ ZOO_DATA_LOG_DIR=/datalog \ ZOO_PORT=2181 \ ZOO_TICK_TIME=2000 \ ZOO_INIT_LIMIT=5 \ ZOO_SYNC_LIMIT=2 # Add a user and make dirs RUN set -x \ && adduser -D "$ZOO_USER" \ && mkdir -p "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR" \ && chown "$ZOO_USER:$ZOO_USER" "$ZOO_DATA_LOG_DIR" "$ZOO_DATA_DIR" "$ZOO_CONF_DIR" ARG GPG_KEY=C823E3E5B12AF29C67F81976F5CECB3CB5E9BD2D ARG DISTRO_NAME=zookeeper-3.4.10 # Download Apache Zookeeper, verify its PGP signature, untar and clean up RUN set -x \ && apk add --no-cache --virtual .build-deps \ gnupg \ && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz" \ && wget -q "http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz.asc" \ && export GNUPGHOME="$(mktemp -d)" \ && gpg --keyserver ha.pool.sks-keyservers.net --recv-key "$GPG_KEY" \ && gpg --batch --verify "$DISTRO_NAME.tar.gz.asc" "$DISTRO_NAME.tar.gz" \ && tar -xzf "$DISTRO_NAME.tar.gz" \ && mv "$DISTRO_NAME/conf/"* "$ZOO_CONF_DIR" \ && rm -r "$GNUPGHOME" "$DISTRO_NAME.tar.gz" "$DISTRO_NAME.tar.gz.asc" \ && apk del .build-deps ADD zkui-master $ZOO_UI_DIR WORKDIR $DISTRO_NAME VOLUME ["$ZOO_DATA_DIR", "$ZOO_DATA_LOG_DIR"] EXPOSE $ZOO_PORT 2888 3888 ENV PATH=$PATH:/$DISTRO_NAME/bin:$ZOO_UI_DIR \ ZOOCFGDIR=$ZOO_CONF_DIR COPY docker-entrypoint.sh / ENTRYPOINT ["/docker-entrypoint.sh"]3，docker-entrypoint.sh如下：
#!/bin/bash set -e # Allow the container to be started with `--user` if [ "$1" = 'zkServer.sh' -a "$(id -u)" = '0' ]; then chown -R "$ZOO_USER" "$ZOO_DATA_DIR" "$ZOO_DATA_LOG_DIR" exec su-exec "$ZOO_USER" "$0" "$@" fi # Generate the config only if it doesn't exist if [ ! -f "$ZOO_CONF_DIR/zoo.cfg" ]; then CONFIG="$ZOO_CONF_DIR/zoo.cfg" echo "clientPort=$ZOO_PORT" >> "$CONFIG" echo "dataDir=$ZOO_DATA_DIR" >> "$CONFIG" echo "dataLogDir=$ZOO_DATA_LOG_DIR" >> "$CONFIG" echo "tickTime=$ZOO_TICK_TIME" >> "$CONFIG" echo "initLimit=$ZOO_INIT_LIMIT" >> "$CONFIG" echo "syncLimit=$ZOO_SYNC_LIMIT" >> "$CONFIG" for server in $ZOO_SERVERS; do echo "$server" >> "$CONFIG" done fi if [ -f "$ZOO_UI_DIR/config.cfg" ]; then CONFIG="$ZOO_UI_DIR/config.cfg" echo "serverPort=$ZOO_UI_PORT" >> "$CONFIG" echo "zkServer=$ZOO_UI_SERVER" >> "$CONFIG" fi # Write myid only if it doesn't exist if [ ! -f "$ZOO_DATA_DIR/myid" ]; then echo "${ZOO_MY_ID:-1}" > "$ZOO_DATA_DIR/myid" fi cd $ZOO_UI_DIR exec nohup java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar & exec zkServer.sh start-foreground[制作过程需要的文件]http://pan.baidu.com/s/1i5IG6sH
执行
➜ wldocker sudo docker build -t zookeeper_3.4.10_zkui_2.0:0.0.1 .就可以制作出包含ZKUI的zookeeper 3.4.10的docker镜像了
![]()
把该镜像上传到HARBOR库，稍后k8s编排时会用到。
- 通过Kubernetes 编排启动Zookeeper集群：
我们采取的方案是每个zkserver属于一个pod，每个pod绑定一个物理主机。
只需要完成yaml文件即可:![]()
zk-1.yaml:
--- apiVersion: v1 kind: Service metadata: name: zk-1-svc labels: app: zk-1-svc spec: ports: - port: 2888 name: server - port: 3888 name: leader-election - port: 2181 name: client - port: 9999 name: zkui selector: app: zk-1 type: NodePort --- apiVersion: v1 kind: ConfigMap metadata: name: zk-1-cm data: jvm.heap: "1G" tick: "2000" init: "10" sync: "5" client.cnxns: "60" snap.retain: "3" purge.interval: "0" --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: zk-1 spec: replicas: 1 template: metadata: labels: app: zk-1 spec: nodeSelector: zk: zk-1 containers: - name: myzk imagePullPolicy: IfNotPresent image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1 resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election - containerPort: 9999 name: zkui env: - name : ZK_HEAP_SIZE valueFrom: configMapKeyRef: name: zk-1-cm key: jvm.heap - name : ZK_TICK_TIME valueFrom: configMapKeyRef: name: zk-1-cm key: tick - name : ZK_INIT_LIMIT valueFrom: configMapKeyRef: name: zk-1-cm key: init - name : ZK_SYNC_LIMIT valueFrom: configMapKeyRef: name: zk-1-cm key: tick - name : ZK_MAX_CLIENT_CNXNS valueFrom: configMapKeyRef: name: zk-1-cm key: client.cnxns - name: ZK_SNAP_RETAIN_COUNT valueFrom: configMapKeyRef: name: zk-1-cm key: snap.retain - name: ZK_PURGE_INTERVAL valueFrom: configMapKeyRef: name: zk-1-cm key: purge.interval - name: ZK_CLIENT_PORT value: "2181" - name: ZK_SERVER_PORT value: "2888" - name: ZK_ELECTION_PORT value: "3888" - name: ZOO_MY_ID value: "1" - name: ZOO_SERVERS value: "server.1=0.0.0.0:2888:3888 server.2=zk-2-svc:2888:3888" - name: ZOO_UI_PORT value: "9999" - name: ZOO_UI_SERVER value: "zk-1-svc:2181,zk-2-svc:2181" # readinessProbe: # exec: # command: # - "zkok.sh" # initialDelaySeconds: 10 # timeoutSeconds: 5 # livenessProbe: # exec: # command: # - "zkok.sh" # initialDelaySeconds: 10 # timeoutSeconds: 5 volumeMounts: - name: data mountPath: /data - name: datalog mountPath: /datalog volumes: - name: data hostPath: path: /data/k8s/zookeeper/data - name: datalog hostPath: path: /data/k8s/zookeeper/datalogzk-2.yaml:
--- apiVersion: v1 kind: Service metadata: name: zk-2-svc labels: app: zk-2-svc spec: ports: - port: 2888 name: server - port: 3888 name: leader-election - port: 2181 name: client - port: 9999 name: zkui selector: app: zk-2 type: NodePort --- apiVersion: v1 kind: ConfigMap metadata: name: zk-2-cm data: jvm.heap: "1G" tick: "2000" init: "10" sync: "5" client.cnxns: "60" snap.retain: "3" purge.interval: "0" --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: zk-2 spec: replicas: 1 template: metadata: labels: app: zk-2 spec: nodeSelector: zk: zk-2 containers: - name: myzk imagePullPolicy: IfNotPresent image: registry.k8s.intra.knownsec.com/bigdata/zookeeper_3.4.10_zkui_2.0:0.0.1 resources: requests: memory: "2Gi" cpu: "500m" ports: - containerPort: 2181 name: client - containerPort: 2888 name: server - containerPort: 3888 name: leader-election - containerPort: 9999 name: zkui env: - name : ZK_HEAP_SIZE valueFrom: configMapKeyRef: name: zk-2-cm key: jvm.heap - name : ZK_TICK_TIME valueFrom: configMapKeyRef: name: zk-2-cm key: tick - name : ZK_INIT_LIMIT valueFrom: configMapKeyRef: name: zk-2-cm key: init - name : ZK_SYNC_LIMIT valueFrom: configMapKeyRef: name: zk-2-cm key: tick - name : ZK_MAX_CLIENT_CNXNS valueFrom: configMapKeyRef: name: zk-2-cm key: client.cnxns - name: ZK_SNAP_RETAIN_COUNT valueFrom: configMapKeyRef: name: zk-2-cm key: snap.retain - name: ZK_PURGE_INTERVAL valueFrom: configMapKeyRef: name: zk-2-cm key: purge.interval - name: ZK_CLIENT_PORT value: "2181" - name: ZK_SERVER_PORT value: "2888" - name: ZK_ELECTION_PORT value: "3888" - name: ZOO_MY_ID value: "2" - name: ZOO_SERVERS value: "server.1=zk-1-svc:2888:3888 server.2=0.0.0.0:2888:3888 " - name: ZOO_UI_PORT value: "9999" - name: ZOO_UI_SERVER value: "zk-1-svc:2181,zk-2-svc:2181" # readinessProbe: # exec: # command: # - "zkok.sh" # initialDelaySeconds: 10 # timeoutSeconds: 5 # livenessProbe: # exec: # command: # - "zkok.sh" # initialDelaySeconds: 10 # timeoutSeconds: 5 volumeMounts: - name: data mountPath: /data - name: datalog mountPath: /datalog volumes: - name: data hostPath: path: /data/k8s/zookeeper/data - name: datalog hostPath: path: /data/k8s/zookeeper/datalog[yaml共享]http://pan.baidu.com/s/1jIyvGPK
执行：
➜ zookeeper kubectl create -f zk-1.yaml service "zk-1-svc" created configmap "zk-1-cm" created deployment "zk-1" created ➜ zookeeper kubectl create -f zk-2.yaml service "zk-2-svc" created configmap "zk-2-cm" created deployment "zk-2" created
- 功能验证：
➜ zookeeper kubectl get po #看po情况 NAME READY STATUS RESTARTS AGE zk-1-1238243890-phcm8 1/1 Running 0 1m zk-2-4022999611-x27vq 1/1 Running 0 1m ➜ zookeeper kubectl get po -o wide #看绑定物理节点情况 NAME READY STATUS RESTARTS AGE IP NODE zk-1-1238243890-phcm8 1/1 Running 0 1m 11.233.112.152 yuyan1 zk-2-4022999611-x27vq 1/1 Running 0 1m 11.233.77.83 yuyan2 ➜ zookeeper kubectl get service #看端口映射情况 NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 11.233.0.1 <none> 443/TCP 4d zk-1-svc 11.233.30.168 <nodes> 2888:31292/TCP,3888:30513/TCP,2181:31570/TCP,9999:30476/TCP 1m zk-2-svc 11.233.27.68 <nodes> 2888:30906/TCP,3888:30842/TCP,2181:32473/TCP,9999:31490/TCP 1m ➜ zookeeper kubectl exec zk-2-4022999611-x27vq -t -i -- bash #进入容器看服务情况 bash-4.3# zkServer.sh status ZooKeeper JMX enabled by default Using config: /conf/zoo.cfg Mode: leader #选举正常![]()
通过宿主机ip加映射后的端口访问zkui http://10.0.8.183:31490
![]()
提供对外访问正常参考
[Kargo部署K8s]http://oo3p.com/2017/04/17/kargo容器化部署kubernetes高可用集群/
[zookeeper docker]https://github.com/31z4/zookeeper-docker/tree/f12428ab7c6ea263ef037cf258129b83276c009c
[k8s pod]https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
[k8s编排zookeeper集群]https://github.com/kubernetes/contrib/tree/master/statefulsets/zookeeper
作者：俺是亮哥
链接：http://www.jianshu.com/p/315e18c2f3a2
來源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。
权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区
更多推荐
1
0- 0
已为社区贡献35条内容
所有评论(0)