zookeeper和k8s_k8s部署有状态服务zookeeper示例

我们先不考虑配置文件的前提下：apiVersion: apps/v1kind: StatefulSet#####固定hostname，有状态的服务使用这个 statefalset有个问题，就是如果那个pod不是running状态，这个主机名是无法解析的，这样就构成了一个死循环，我sed替换主机名的时候由于pod还不是running状态，她只能获取自己的主机名。无法获取别人的主机名，所以在zo...

wdk199512

706人浏览 · 2020-12-22 04:20:25

wdk199512 · 2020-12-22 04:20:25 发布

我们先不考虑配置文件的前提下：

apiVersion: apps/v1

kind: StatefulSet #####固定hostname，有状态的服务使用这个 statefalset有个问题，就是如果那个pod不是running状态，这个主机名是无法解析的，这样就构成了一个死循环，我sed替换主机名的时候由于pod还不是running状态，她只能获取自己的主机名。无法获取别人的主机名，所以在zookeeper中换成了换成了ip

metadata:

spec:

serviceName: zookeeper ####所以生成的3个pod的名字叫zookeeper-0,zookeeper-1,zookeeper-2

replicas: 3

revisionHistoryLimit: 10

selector: ##statefulset必须有的

matchLabels:

app: zookeeper

template:

metadata:

labels:

app: zookeeper

spec:

volumes:

- name: volume-logs

hostPath:

path: /var/log/zookeeper

containers:

- name: zookeeper

image: harbor.test.com/middleware/zookeeper:3.4.10

imagePullPolicy: IfNotPresent

livenessProbe:

tcpSocket:

port: 2181

initialDelaySeconds: 30

timeoutSeconds: 3

periodSeconds: 5

successThreshold: 1

failureThreshold: 2

ports:

- containerPort: 2181

protocol: TCP

- containerPort: 2888

protocol: TCP

- containerPort: 3888

protocol: TCP

env:

- name: SERVICE_NAME

value: "zookeeper"

- name: MY_POD_NAME #声明k8s自带的变量，这样在pod创建之后，在其中可以直接echo ${MY_POD_NAME}得到hostname

valueFrom:

fieldRef:

fieldPath: metadata.name

volumeMounts:

- name: volume-logs

mountPath: /var/log/zookeeper

nodeSelector:

zookeeper: enable

---

apiVersion: v1

kind: Service

metadata:

name: zookeeper #我的cluster名字为这个，在任意一个生成的pod中可以ping zookeeper，相当于zookeeper为生成的3个pod的cluster_name,会发现每次ping出的地址不一定相同，nslookup zookeeper得到的是3个pod的pod ip，共3条记录。

spec:

ports:

- port: 2181

selector:

app: zookeeper

clusterIP: None #此句必须加上

[root@host5 src]# kubectl get pod --all-namespaces -o wide

NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

default zookeeper-0 1/1 Running 0 12m 192.168.55.69 host3

default zookeeper-1 1/1 Running 0 12m 192.168.31.93 host4

default zookeeper-2 1/1 Running 0 12m 192.168.55.70 host3

bash-4.3# nslookup zookeeper

nslookup: can't resolve '(null)': Name does not resolve

Name: zookeeper

Address 1: 192.168.55.70 zookeeper-2.zookeeper.default.svc.cluster.local

Address 2: 192.168.55.69 zookeeper-0.zookeeper.default.svc.cluster.local

Address 3: 192.168.31.93 zookeeper-1.zookeeper.default.svc.cluster.local

bash-4.3# ping zookeeper-0.zookeeper

PING zookeeper-0.zookeeper (192.168.55.69): 56 data bytes

64 bytes from 192.168.55.69: seq=0 ttl=63 time=0.109 ms

64 bytes from 192.168.55.69: seq=1 ttl=63 time=0.212 ms

--- zookeeper-0.zookeeper ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 0.109/0.160/0.212 ms

bash-4.3# ping zookeeper-1.zookeeper

PING zookeeper-1.zookeeper (192.168.31.93): 56 data bytes

64 bytes from 192.168.31.93: seq=0 ttl=62 time=0.535 ms

64 bytes from 192.168.31.93: seq=1 ttl=62 time=0.507 ms

64 bytes from 192.168.31.93: seq=2 ttl=62 time=0.587 ms

--- zookeeper-1.zookeeper ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.507/0.543/0.587 ms

bash-4.3# ping zookeeper-2.zookeeper

PING zookeeper-2.zookeeper (192.168.55.70): 56 data bytes

64 bytes from 192.168.55.70: seq=0 ttl=64 time=0.058 ms

64 bytes from 192.168.55.70: seq=1 ttl=64 time=0.081 ms

--- zookeeper-2.zookeeper ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 0.058/0.069/0.081 ms

k8s自带的常用变量如下：

env:

- name: MY_NODE_NAME

valueFrom:

fieldRef:

fieldPath: spec.nodeName

- name: MY_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: MY_POD_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

- name: MY_POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

- name: MY_POD_SERVICE_ACCOUNT

valueFrom:

fieldRef:

fieldPath: spec.serviceAccountName

spec.nodeName ： pod所在节点的IP、宿主主机IP

status.podIP ：pod IP

我们再看配置文件：

[root@docker06 conf]# cat zoo.cfg |grep -v ^#|grep -v ^$

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data

clientPort=2181

clientPortAddress= docker06

server.1=docker05:2888:3888

server.2=docker06:2888:3888

server.3=docker04:2888:3888

snapCount=10000

leaderServes=yes

autopurge.snapRetainCount=3

autopurge.purgeInterval=2

maxClientCnxns=1000

我们需要修改成形如：

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data

clientPort=2181

clientPortAddress= docker06 #下面的3行是固定的，主要是这行需要修改成本机的MY_POD_IP,我们可以用configmap挂载配置文件，然后在pod里面用sed替换掉这行配置

server.1=zookeeper-0.zookeeper:2888:3888

server.2=zookeeper-1.zookeeper:2888:3888

server.3=zookeeper-2.zookeeper:2888:3888

snapCount=10000

leaderServes=yes

autopurge.snapRetainCount=3

autopurge.purgeInterval=2

maxClientCnxns=1000

参考如下这种方式：

先将配置文件通过configmap挂载进pod里面,如fix-ip.sh

apiVersion: v1

kind: ConfigMap

metadata:

labels:

app: redis-cluster

data:

fix-ip.sh: |

#!/bin/sh

CLUSTER_CONFIG="/var/lib/redis/nodes.conf"

if [ -f ${CLUSTER_CONFIG} ]; then

if [ -z "${POD_IP}" ]; then

echo "Unable to determine Pod IP address!"

exit 1

echo "Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}"

sed -i.bak -e "/myself/s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${CLUSTER_CONFIG}

exec "$@"

redis.conf: |+

cluster-enabled yes

cluster-require-full-coverage no

cluster-node-timeout 15000

cluster-config-file /var/lib/redis/nodes.conf

cluster-migration-barrier 1

appendonly yes

protected-mode no

然后在启动pod的时候执行这个脚本：

apiVersion: apps/v1

kind: StatefulSet

metadata:

labels:

app: redis-cluster

spec:

serviceName: redis-cluster

replicas: 6

selector:

matchLabels:

app: redis-cluster

template:

metadata:

labels:

app: redis-cluster

spec:

containers:

- name: redis

image: 10.11.100.85/library/redis

ports:

- containerPort: 6379

- containerPort: 16379

command: ["/etc/redis/fix-ip.sh", "redis-server", "/etc/redis/redis.conf"] #此处先执行了那个脚本，然后启动的redis

readinessProbe:

exec:

command:

- sh

- -c

- "redis-cli -h $(hostname) ping"

initialDelaySeconds: 15

timeoutSeconds: 5

livenessProbe:

exec:

command:

- sh

- -c

- "redis-cli -h $(hostname) ping"

initialDelaySeconds: 20

periodSeconds: 3

env:

- name: POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

volumeMounts:

- name: conf

mountPath: /etc/redis

readOnly: false

- name: data

mountPath: /var/lib/redis

readOnly: false

volumes:

- name: conf

configMap:

defaultMode: 0755

# items:

# - key: redis.conf

# path: redis.conf

# - key: fix-ip.sh

# path: fix-ip.sh

volumeClaimTemplates:

- metadata:

labels:

spec:

accessModes: [ "ReadWriteOnce" ]

resources:

requests:

storage: 150Mi

注意：通过configmap生成的配置文件为只读，无法通过sed修改，可以通过挂载到临时目录，然后拷过去之后sed，但是这样也存在一个问题，就是你动态修改了configmap，只会改变临时目录里的文件，而不会改变考过去的文件

实际生产环境的配置：

1.重新制作image

[root@host4 zookeeper]# ll

总用量 4

drwxr-xr-x 2 root root 45 5月 24 15:48 conf

-rw-r--r-- 1 root root 143 5月 23 06:19 Dockerfile

drwxr-xr-x 2 root root 20 5月 24 15:48 scripts

[root@host4 conf]# cd conf

[root@host4 conf]# ll

总用量 8

-rw-r--r-- 1 root root 1503 5月 23 04:15 log4j.properties

-rw-r--r-- 1 root root 324 5月 24 15:48 zoo.cfg

[root@host4 conf]# cat zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data

clientPort=2181

clientPortAddress=PODIP #此处用ip，下面用主机名，原因见本文上面

server.1=zookeeper-0.zookeeper:2888:3888

server.2=zookeeper-1.zookeeper:2888:3888

server.3=zookeeper-2.zookeeper:2888:3888

snapCount=10000

leaderServes=yes

autopurge.snapRetainCount=3

autopurge.purgeInterval=2

maxClientCnxns=1000

[root@host4 conf]# cd ../scripts/

[root@host4 scripts]# ll

总用量 4

-rwxr-xr-x 1 root root 177 5月 24 15:48 sed.sh

[root@host4 scripts]# cat sed.sh

#!/bin/bash

MY_ID=`echo ${MY_POD_NAME} |awk -F'-' '{print $NF}'`

MY_ID=`expr ${MY_ID} + 1`

echo ${MY_ID} > /data/myid

sed -i 's/PODIP/'${MY_POD_IP}'/g' /conf/zoo.cfg

exec "$@"

[root@host4 scripts]# cd ..

[root@host4 zookeeper]# ls

conf Dockerfile scripts

[root@host4 zookeeper]# cat Dockerfile

FROM harbor.test.com/middleware/zookeeper:3.4.10

MAINTAINER rongruixue@163.com

ARG zookeeper_version=3.4.10

COPY conf /conf/

COPY scripts /

这样我们docker build就制作出了image ：harbor.test.com/middleware/zookeeper:v3.4.10

然后我们启通过yml启动pod：

apiVersion: apps/v1

kind: StatefulSet

metadata:

spec:

# podManagementPolicy: Parallel #此配置决定是否让3个pod同时起来，而不是按 0 1 2的顺序

serviceName: zookeeper

replicas: 3

revisionHistoryLimit: 10

selector:

matchLabels:

app: zookeeper

template:

metadata:

labels:

app: zookeeper

spec:

volumes:

- name: volume-logs

hostPath:

path: /var/log/zookeeper

- name: volume-data

hostPath:

path: /opt/zookeeper/data

terminationGracePeriodSeconds: 10

containers:

- name: zookeeper

image: harbor.test.com/middleware/zookeeper:v3.4.10

imagePullPolicy: Always

ports:

- containerPort: 2181

protocol: TCP

- containerPort: 2888

protocol: TCP

- containerPort: 3888

protocol: TCP

env:

- name: SERVICE_NAME

value: "zookeeper"

- name: MY_POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

- name: MY_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

volumeMounts:

- name: volume-logs

mountPath: /var/log/zookeeper

#- name: volume-data 此处不能挂载、data到本地，否则如果两个pod分配到同一个节点会相互覆盖，myid也会被覆盖

# mountPath: /data

command:

- /bin/bash

- -c

- -x

- |

/sed.sh #此脚本作用就是讲podip写入zoo.cfg配置文件中，然后写/data/myid

sleep 10

zkServer.sh start-foreground

nodeSelector:

zookeeper: enable

---

apiVersion: v1

kind: Service

metadata:

spec:

ports:

- port: 2181

selector:

app: zookeeper

clusterIP: None

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub