【K8S、Redis、集群】kubernetes中部署redis集群三主三从
在kubernetes中部署三主三从的redis cluster。PV/PVC的动态供给有状态pod控制器StatefulSet和headless servicepod每次重启ip都会变化的问题redis集群因为重定向的问题,不能从k8s外部访问如何解决这里使用动态卷,不需要提前创建好PV和PVC,而是由nfs-client插件自动创建。2.准备redis镜像测试下镜像是否可以正常运行,不报错:3
·
前言
在kubernetes中部署三主三从的redis cluster。
知识点
-
PV/PVC的动态供给
-
有状态pod控制器StatefulSet和headless service
-
pod每次重启ip都会变化的问题
-
redis集群因为重定向的问题,不能从k8s外部访问如何解决
1. 准备存储环境
这里使用动态卷,不需要提前创建好PV和PVC,而是由nfs-client插件自动创建。
关于动态卷供给:
https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
执行步骤
#pwd
/data/kube_yaml/deploy_nfs_storageclass
# ls -l
-rw-r--r-- 1 root root 245 Feb 2 22:09 class.yaml
-rw-r--r-- 1 root root 1075 Feb 2 22:09 deployment.yaml
-rw-r--r-- 1 root root 1900 Feb 2 22:09 rbac.yaml
# 修改里面对应的namespace
# kubectl apply -f rbac.yaml
# 修改里面对应的namespace
# kubectl apply -f deployment.yaml
#这个是全局的,如果以前执行过,就不需要再执行了,但是可以重复执行,所以执行一下也没问题
# kubectl apply -f class.yaml
2.准备redis镜像
# pwd
/data/kube_yaml/redis/dockerfile
# ls -l
-rwxr-xr-x 1 root root 96 Feb 2 23:02 build-command.sh
-rw-r--r-- 1 root root 296 Feb 2 23:10 Dockerfile
-rw-r--r-- 1 root root 2000179 Jan 31 14:44 redis-5.0.14.tar.gz #这个源码包是在redis官方下载的
-rw-r--r-- 1 root root 248 Feb 2 23:11 redis.conf
# cat Dockerfile
FROM harbor.fanheng.com/baseimages/fanheng-centos-base:7.9.2009
ADD redis-5.0.14.tar.gz /usr/local/src/
RUN mkdir -pv /opt/redis-data /usr/local/redis
COPY redis.conf /usr/local/redis/
RUN cd /usr/local/src/redis-5.0.14 && make && make install
CMD ["redis-server","/usr/local/redis/redis.conf"]
# cat redis.conf
appendonly yes
protected-mode no
port 6379
bind 0.0.0.0
requirepass 123456
save 3600 1 300 100 60 10000
dir /opt/redis-data
dbfilename dump.rdb
#logfile redis.log
#pidfile redis.pid
masterauth 123456
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
# cat build-command.sh
#!/bin/bash
tag="harbor.fanheng.com/fanheng/redis:$1"
docker build -t $tag .
docker push $tag
# bash build-command.sh 5.0.14
测试下镜像是否可以正常运行,不报错:
# docker run -it --rm harbor.fanheng.com/fanheng/redis:5.0.14
3.在k8s中创建redis集群
# cat redis.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster-config
namespace: fanheng
data:
redis-cluster-configmap: |
appendonly yes
protected-mode no
port 6379
bind 0.0.0.0
requirepass 123456
save 3600 1 300 100 60 10000
dir /opt/redis-data
dbfilename dump.rdb
#logfile redis.log
#pidfile redis.pid
masterauth 123456
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
---
kind: Service
apiVersion: v1
metadata:
labels:
app: redis-cluster-headless
name: redis-cluster-headless
namespace: fanheng
spec:
clusterIP: None
ports:
- name: redis
port: 6379
protocol: TCP
targetPort: 6379
- name: redis-cluster
port: 16379
protocol: TCP
targetPort: 16379
selector:
app: redis-cluster
---
kind: Service
apiVersion: v1
metadata:
labels:
app: redis-cluster-access
name: redis-cluster-access
namespace: fanheng
spec:
type: NodePort
ports:
- name: http
port: 6379
protocol: TCP
targetPort: 6379
nodePort: 56379
selector:
app: redis-cluster
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: redis-cluster
name: redis-cluster
namespace: fanheng
spec:
serviceName: redis-cluster-headless
replicas: 6
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
spec:
spec:
terminationGracePeriodSeconds: 20
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis-cluster
topologyKey: kubernetes.io/hostname
containers:
- name: redis-cluster
image: harbor.fanheng.com/fanheng/redis:5.0.14
command:
- /bin/bash
- -c
- |
HOSTNAME=$(hostname)
echo $HOSTNAME
echo $MY_POD_IP
#sleep 20 #这里之所以要睡20秒,是因为cluster-node-timeout 15000这里需要等待15秒才会接管master,而在15秒内master pod恢复了,那么之前未被RDB持久化的数据就会丢失。配置了AOF这里就不需要了,因为AOF默认每秒异步记录一次变化日志,但是建议这里加上。
redis-server /usr/local/redis/redis.conf --protected-mode no --cluster-announce-ip $(MY_POD_IP)
env:
- name: MY_POD_IP # 用于pod重启时获取pod新的ip,--cluster-announce-ip $(MY_POD_IP)用于更新redis集群的ip
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 6379
name: redis
- containerPort: 16379
name: redis-cluster
resources:
requests:
cpu: "500m"
memory: "500Mi"
volumeMounts:
- name: redis-data
mountPath: /opt/redis-data
- name: redis-cluster-config
mountPath: /usr/local/redis
volumes:
- name: redis-cluster-config
configMap:
name: redis-cluster-config
items:
- key: redis-cluster-configmap
path: redis.conf
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
storageClassName: "nfs-client"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
# kubectl apply -f redis.yaml
执行集群初始化
# kubectl get pods -n fanheng
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5f684db56b-85z5c 1/1 Running 0 98m
redis-cluster-0 1/1 Running 0 5m26s
redis-cluster-1 1/1 Running 0 5m23s
redis-cluster-2 1/1 Running 0 5m20s
redis-cluster-3 1/1 Running 0 5m17s
redis-cluster-4 1/1 Running 0 5m14s
redis-cluster-5 1/1 Running 0 5m11s
需要登录到一个任意redis节点执行,这里选择第一个
# kubectl exec -it -n fanheng redis-cluster-0 -- bash
# yum -y install bind-utils
# redis-cli --cluster create \
`dig +short redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
`dig +short redis-cluster-1.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
`dig +short redis-cluster-2.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
`dig +short redis-cluster-3.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
`dig +short redis-cluster-4.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
`dig +short redis-cluster-5.redis-cluster-headless.fanheng.svc.fanheng.local`:6379 \
--cluster-replicas 1 \
-a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.200.166.242:6379 to 10.200.150.66:6379
Adding replica 10.200.150.69:6379 to 10.200.166.232:6379
Adding replica 10.200.150.68:6379 to 10.200.37.152:6379
M: 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7 10.200.150.66:6379
slots:[0-5460] (5461 slots) master
M: 8a71876eb50e553cb79480f9fefcb505fcf251f5 10.200.166.232:6379
slots:[5461-10922] (5462 slots) master
M: cab5f390c61b2faf613c5eff533772dfb7ea6088 10.200.37.152:6379
slots:[10923-16383] (5461 slots) master
S: 78a248c46fb503bd93664374dbf3e4943043a2cf 10.200.150.68:6379
replicates cab5f390c61b2faf613c5eff533772dfb7ea6088
S: 561f0063c5d72b81d9ac1c0e7f2af82aaa746efe 10.200.166.242:6379
replicates 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7
S: 0b67723f3657467d7db4f525d9124b377ede8906 10.200.150.69:6379
replicates 8a71876eb50e553cb79480f9fefcb505fcf251f5
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 10.200.150.66:6379)
M: 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7 10.200.150.66:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 8a71876eb50e553cb79480f9fefcb505fcf251f5 10.200.166.232:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 0b67723f3657467d7db4f525d9124b377ede8906 10.200.150.69:6379
slots: (0 slots) slave
replicates 8a71876eb50e553cb79480f9fefcb505fcf251f5
S: 561f0063c5d72b81d9ac1c0e7f2af82aaa746efe 10.200.166.242:6379
slots: (0 slots) slave
replicates 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7
M: cab5f390c61b2faf613c5eff533772dfb7ea6088 10.200.37.152:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 78a248c46fb503bd93664374dbf3e4943043a2cf 10.200.150.68:6379
slots: (0 slots) slave
replicates cab5f390c61b2faf613c5eff533772dfb7ea6088
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
检查集群
# redis-cli -a 123456 --cluster check redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379 (25a0eaf7...) -> 4 keys | 5461 slots | 1 slaves.
10.200.166.232:6379 (8a71876e...) -> 2 keys | 5462 slots | 1 slaves.
10.200.37.152:6379 (cab5f390...) -> 4 keys | 5461 slots | 1 slaves.
[OK] 10 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379)
M: 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7 redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 561f0063c5d72b81d9ac1c0e7f2af82aaa746efe 10.200.166.242:6379
slots: (0 slots) slave
replicates 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7
M: 8a71876eb50e553cb79480f9fefcb505fcf251f5 10.200.166.232:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 78a248c46fb503bd93664374dbf3e4943043a2cf 10.200.150.68:6379
slots: (0 slots) slave
replicates cab5f390c61b2faf613c5eff533772dfb7ea6088
S: 0b67723f3657467d7db4f525d9124b377ede8906 10.200.150.69:6379
slots: (0 slots) slave
replicates 8a71876eb50e553cb79480f9fefcb505fcf251f5
M: cab5f390c61b2faf613c5eff533772dfb7ea6088 10.200.37.152:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# redis-cli -c -h redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:11781
cluster_stats_messages_pong_sent:11187
cluster_stats_messages_sent:22968
cluster_stats_messages_ping_received:11187
cluster_stats_messages_pong_received:11781
cluster_stats_messages_received:22968
redis-cluster-0.redis-cluster-headless.fanheng.svc.fanheng.local:6379> cluster nodes
561f0063c5d72b81d9ac1c0e7f2af82aaa746efe 10.200.166.242:6379@16379 slave 25a0eaf7b0bc64c78649e24c2e008e64bc979bf7 0 1675454105000 5 connected
25a0eaf7b0bc64c78649e24c2e008e64bc979bf7 10.200.150.127:6379@16379 myself,master - 0 1675454105000 1 connected 0-5460
8a71876eb50e553cb79480f9fefcb505fcf251f5 10.200.166.232:6379@16379 master - 0 1675454104000 2 connected 5461-10922
78a248c46fb503bd93664374dbf3e4943043a2cf 10.200.150.68:6379@16379 slave cab5f390c61b2faf613c5eff533772dfb7ea6088 0 1675454106000 4 connected
0b67723f3657467d7db4f525d9124b377ede8906 10.200.150.69:6379@16379 slave 8a71876eb50e553cb79480f9fefcb505fcf251f5 0 1675454105000 6 connected
cab5f390c61b2faf613c5eff533772dfb7ea6088 10.200.37.152:6379@16379 master - 0 1675454107003 3 connected 10923-16383
注意
此时只能k8s集群内部访问,不能在k8s集群外部访问。原因:因为通过service访问redis集群时,redis集群会把访问重定向到k8s内部的podIP.
# redis-cli -c -h 192.168.88.13 -p 56379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.88.13:56379> get k1
-> Redirected to slot [12706] located at 10.200.37.152:6379
这里的10.200.37.152是k8s内部的ip,所以从k8s外部访问不到
4.如何从k8s外部访问redis集群
需要用到redis官方提供的proxy插件
官方文档:
https://github.com/RedisLabs/redis-cluster-proxy
使用总结
编译redis-proxy
# git clone https://github.com/RedisLabs/redis-cluster-proxy
# cd redis-cluster-proxy/
# make PREFIX=/usr/local/redis_cluster_proxy install
然后会生成一个二进制文件:
/usr/local/redis_cluster_proxy/bin/redis-cluster-proxy
把这个二进制文件拷贝到Dockerfile所在目录:
# ls -l
-rw-r--r-- 1 root root 112 Feb 4 01:20 build-command.sh
-rw-r--r-- 1 root root 122 Feb 4 02:40 Dockerfile
-rwxr-xr-x 1 root root 1172736 Feb 4 01:15 redis-cluster-proxy
-rw-r--r-- 1 root root 1569 Feb 4 01:26 redis-cluster-proxy.yaml
制作redis-proxy镜像
# cat build-command.sh
#!/bin/bash
tag="harbor.fanheng.com/baseimages/redis-cluster-proxy:v1"
docker build . -t $tag
docker push $tag
root@harbor1:/data/kube_ya
创建pod
# cat redis-cluster-proxy.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-proxy
namespace: fanheng
data:
proxy.conf: |
cluster redis-cluster-headless:6379 # 配置为Redis Cluster Service
bind 0.0.0.0
port 7777 # redis-cluster-proxy 对外暴露端口
threads 8 # 线程数量
daemonize no
enable-cross-slot yes
auth 123456 # 配置Redis Cluster 认证密码
log-level error
---
apiVersion: v1
kind: Service
metadata:
name: redis-proxy
namespace: fanheng
spec:
type: NodePort # 对K8S外部提供服务
ports:
- name: redis-proxy
nodePort: 37777 # 对外提供的端口
port: 7777
protocol: TCP
targetPort: 7777
selector:
app: redis-proxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-proxy
namespace: fanheng
spec:
replicas: 1
selector:
matchLabels:
app: redis-proxy
template:
metadata:
labels:
app: redis-proxy
spec:
containers:
- name: redis-proxy
image: harbor.fanheng.com/baseimages/redis-cluster-proxy:v1
imagePullPolicy: Always
command: ["redis-cluster-proxy"]
args:
- -c
- /data/proxy.conf # 指定启动配置文件
ports:
- name: redis-7777
containerPort: 7777
protocol: TCP
volumeMounts:
- name: redis-proxy-conf
mountPath: /data/
volumes: # 挂载proxy配置文件
- name: redis-proxy-conf
configMap:
name: redis-proxy
# kubectl apply -f redis-cluster-proxy.yaml
# kubectl get pod -n fanheng
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 14 (34m ago) 26h
nfs-client-provisioner-5f684db56b-85z5c 1/1 Running 1 (7h34m ago) 28h
redis-cluster-0 1/1 Running 0 128m
redis-cluster-1 1/1 Running 0 132m
redis-cluster-2 1/1 Running 0 132m
redis-cluster-3 1/1 Running 0 132m
redis-cluster-4 1/1 Running 0 132m
redis-cluster-5 1/1 Running 0 132m
redis-proxy-996946765-kcscp 1/1 Running 0 4m30s
从外部访问redis-proxy的svc
# kubectl get svc -n fanheng
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-cluster-access NodePort 10.100.68.95 <none> 6379:56379/TCP 130m
redis-cluster-headless ClusterIP None <none> 6379/TCP,16379/TCP 130m
redis-proxy NodePort 10.100.73.169 <none> 7777:37777/TCP 2m13s
# redis-cli -h 192.168.88.13 -p 37777 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.88.13:37777> get k1
"v1"
192.168.88.13:37777> get k2
"v2"
192.168.88.13:37777> get k3
"v3"
192.168.88.13:37777> get k4
"v4"
192.168.88.13:37777> get k5
"v5"
192.168.88.13:37777> get k6
"v6"
192.168.88.13:37777> get k7
"v7"
192.168.88.13:37777> get k8
"v8"
python访问测试:
import redis
r = redis.Redis(
host="192.168.88.13",
port="37777",
password="123456",
charset="utf8",
decode_responses=True
)
res = r.get("k1")
print(res)
更多推荐
已为社区贡献1条内容
所有评论(0)