前言

自己编写k8s部署文件比较麻烦,成熟的开源软件的k8s部署文件如redis,mysql并不需要自己从头编写,只需要通过一个类似yum的部署工具heml,
将heml官方或其它第三方或私有heml仓库拉下来,或稍作自定义修改就可以在k8s上跑起来。但自己的业务系统还是得自己编写k8s部署文件或helm chart（需要掌握helm的模板语法等）进行部署,
这里先通过自己写部署文件部署个redis,再通过helm部署一个

自己写部署文件redis-deployment-service.yml

说明：这里是更具docker-compose文件自己翻译过来的，这里用的Deployment,存储卷用的hostpath不好合适，最好的方案的使用statefulSet+nfs/glusterfs等持久卷（后面的heml安装即用该方式），
由于使用hostpath挂载redis的data,固使用了node打label方式让redis pod只调度到指定的node

  • 1.准备 kubectl create namespace jwolf ; kubectl label node node1 mynodelable=node1 ; kubectl label node node2 mynodelable=node2
  • 2.先创建configmap用于存放redis配置: kubectl create configmap redis-configmap --from-file=/etc/redis/redis.conf -n jwolf
  • 3.创建：kubectl apply -f redis-deployment-service.yml
  • 4.查看各资源状态：kubectl get all -A -o wide

kind: Service
apiVersion: v1
metadata:
  labels:
    name: redis
  name: redis
  namespace: jwolf    #使用自定义namespace
spec:
  type: NodePort 
  ports:
    - name: redis
      port: 6379
      targetPort: 6379
      nodePort: 6379
  selector:
    name: redis
---
apiVersion: apps/v1  
kind: Deployment
metadata:
  name: redis
  namespace: jwolf    #使用自定义namespace
spec:
  replicas: 1 
  selector:
    matchLabels:
      name: redis
  template:
    metadata:
      labels:
         name: redis
    spec:
      nodeSelector:
        mynodelable: node1
      containers:
      - name: redis-container
        image: redis:6.2.6
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: myconfigmap     #与volumes的name对应
          mountPath: "/usr/local/etc"   #与vomums的key为redis.conf的path拼接得到容器内redis配置文件绝对路径
        - name: redis-data             #容器内挂载点名称
          mountPath: /data    #容器内data保存路径，该路径在redis.conf配置了
        command:
          - "redis-server"   #启动命令
        args:
          - "/usr/local/etc/redis/redis.conf"  #带配置文件启动
        ports:
             - containerPort: 6379
               name: redis
               protocol: TCP
      volumes:
      - name: myconfigmap
        configMap:
          name: redis-configmap  #指定使用的configmap
          items:
            - key: redis.conf  #该值为configmap创建--from-file的文件名
              path: redis/redis.conf
      - name: redis-data          #数据卷名称，需要与容器内挂载点名称一致
        hostPath:
           path: /home/data/redis-data   #将容器挂载点名称相同的path挂载到宿主机path（但这种方式有局限性，pod分配到其它node数据就丢失了，需要挂载到NFS文件系统或限定pod调度）

使用k8s pv,pvc优化文件挂载,这里使用nfs,先安装先nsf

1.随便搞1个/多个node安装nfs,这里就选个master(192.168.1.18)就够了
yum -y install nfs-utils
2.建两个文件夹作为挂载点（pv2后面helm安装redis1主1从时用）
mkdir /home/data/k8s-data/redis/datadir-pv-1 -p
mkdir /home/data/k8s-data/redis/datadir-pv-2 -p 
3.配置nfs
echo "/home/data/k8s-data/redis/datadir-pv-1   192.168.1.0/24(rw,no_root_squash)"  >> /etc/exports
echo "/home/data/k8s-data/redis/datadir-pv-1   192.168.1.0/24(rw,no_root_squash)"  >> /etc/exports
4.启动并验证
systemctl enable nfs
systemctl start nfs
systemctl status nfs
showmount -e 192.168.1.18

先kubectl apply -f redis-pv-pvc.yaml，待与pod绑定


apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: jwolf
  name: redis-datadir-pv-1
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.1.18
    path: /home/data/k8s-data/redis/datadir-pv-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-datadir-pvc-1
  namespace: jwolf
spec:
  accessModes:
    - ReadWriteMany
  volumeName: redis-datadir-pv-1
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  namespace: jwolf
  name: redis-datadir-pv-2
spec:
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.1.18
    path: /home/data/k8s-data/redis/datadir-pv-2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-datadir-pvc-2
  namespace: jwolf
spec:
  accessModes:
    - ReadWriteMany
  volumeName: redis-datadir-pv-2
  resources:
    requests:
      storage: 2Gi


[root@master k8s]# kubectl get pv,pvc -A -o wide
NAME                                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE   VOLUMEMODE
persistentvolume/redis-datadir-pv-1   4Gi        RWX            Retain           Bound    jwolf/redis-datadir-pvc-1                           3s    Filesystem
persistentvolume/redis-datadir-pv-2   4Gi        RWX            Retain           Bound    jwolf/redis-datadir-pvc-2                           3s    Filesystem

NAMESPACE   NAME                                        STATUS   VOLUME               CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
jwolf       persistentvolumeclaim/redis-datadir-pvc-1   Bound    redis-datadir-pv-1   4Gi        RWX                           3s    Filesystem
jwolf       persistentvolumeclaim/redis-datadir-pvc-2   Bound    redis-datadir-pv-2   4Gi        RWX                           2s    Filesystem

修改redis-deployment-service.yml 的hostPath为pvc(persistentVolumeClaim持久卷申领)，再apply一下即可

      - name: redis-data          #数据卷名称，需要与容器内挂载点名称一致
        persistentVolumeClaim:
          claimName: redis-datadir-pvc-1
       # hostPath:
       #    path: /home/data/redis-data   #将容器挂载点名称相同的path挂载到宿主机path（但这种方式有局限性，pod分配到其它node数据就丢失了，需要挂载到NFS文件系统或限定pod调度）

清理战场kubectl delete -f redis-deployment-service.yml，改用helm来安装一个1主1从的redis集群

1. 下载解压
wget https://get.helm.sh/helm-v3.7.2-linux-amd64.tar.gz 或github 项目releases下载并解压
mv helm /usr/local/bin/
2. 类似yum方式使用,添加一些第三方源
 helm repo add bitnami https://charts.bitnami.com/bitnami
 helm repo add apphub https://apphub.aliyuncs.com
 helm repo add stable http://mirror.azure.cn/kubernetes/charts/    
 helm repo add aliyun  https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
 helm repo list
3.拉取chart并适当修改,官方搜索helm search hub redis,第三方仓库搜索heml search repo redis，helm提供了一个类似dockerhub镜像网站的helm chart网站https://artifacthub.io
建议在网站搜索相关chart进行安装——有足够的chart文档说明。如果不需要自定义参数或个别自定义参数可以直接helm install my-release --set master.persistence.existingClaim=PVC_NAME bitnami/redis进行在线安装，如果自定义配置较多建议
helm pull bitnami/redis 解压后修改里面的valuse.yaml，改好后执行安装离线安装heml install my-release ./my-releasedir -n xxx ,helm chart其实就是别人基于一种模板编写的各种部署并有各种配置参数可以通过修改chart解压包里的
values.yaml进行自定义
4.两个坑：不改配置直接启动helm  helm install  redis ./redis(chart解压目录)   --set replica.replicaCount=1  -n jwolf，其实并不能启动，容器会一直pendding,
kubectl logs -f --tail 222 pod/redis-master-0 -n jwolf 并无日志说明容器还没确定就异常了，需要kubectl describe  pod/redis-master-0 -n jwolf 看到
报错信息【Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "redis-master-0": pod has unbound immediate PersistentVolumeClaims】，bitnami做的这个chart并无默认的
数据卷配置，需要自己bond pvc,修改values.yaml即可或直接commandline指定pvc,先卸了之前的helm uninstall redis -n jwolf，再安装 helm install   redis ./redis  
 --set replica.replicaCount=1,master.persistence.existingClaim=redis-datadir-pvc-1,replica.persistence.enabled=false -n jwolf ,因replica绑定pvc与master还不同，暂未找到绑定方式
 固replica暂时禁用pvc,使用的是临时卷emptyDir

5.最终启动命令
helm install   redis ./redis   --set replica.replicaCount=1,master.persistence.existingClaim=redis-datadir-pvc-1,replica.persistence.enabled=false,master.service.nodePort=6379,master.service.type=NodePort,auth.password='jwolf' -n jwolf
pod状态为CrashLoopBackOff，kubectl logs -f pod/redis-master-0 -n jwolf报错【 Can't open the append-only file: Permission denied】，挂在点加个写权限即可chmod 664 /home/data/k8s-data/redis/datadir-pv-1/* -R

验证

进入容器验证,无论走IP还是DNS，主从能互相访问说明集群通信就OK了

kubectl exec -it pod/redis-master-0 bash -n jwolf

curl redis-master.jwolf.svc.cluster.local:6379 域名拼接规则:Service的metadata.name+命名空间+svc.cluster.local

curl redis-master-0:6379

curl service ClusterIP:6379

curl pod IP:6379

redis-cli -a jwolf

redis-cli -h redis-replicas.jwolf.svc.cluster.local -p 6379 -a jwolf

redis-cli -h redis-master.jwolf.svc.cluster.local -p 6379 -a jwolf

用宿主机的redis-cli直接连接，需要确定redis pod被调度的ip并且宿主机要有redis-cli可执行二进制文件

redis-cli -h 192.168.1.20 -p 6379 -a jwolf

##使用helm输入提示进行验证,

** Please be patient while the chart is being deployed **

Redis™ can be accessed on the following DNS names from within your cluster:

redis-master.jwolf.svc.cluster.local for read/write operations (port 6379)
redis-replicas.jwolf.svc.cluster.local for read-only operations (port 6379)

To get your password run:

export REDIS_PASSWORD=$(kubectl get secret --namespace jwolf redis -o jsonpath="{.data.redis-password}" | base64 --decode)

To connect to your Redis™ server:

  1. Run a Redis™ pod that you can use as a client:

    kubectl run --namespace jwolf redis-client --restart=‘Never’ --env REDISCLI_AUTH=$REDIS_PASSWORD --image docker.io/bitnami/redis:6.2.6-debian-10-r53 --command – sleep infinity

    Use the following command to attach to the pod:

    kubectl exec --tty -i redis-client
    –namespace jwolf – bash

  2. Connect using the Redis™ CLI:
    REDISCLI_AUTH=“ R E D I S P A S S W O R D " r e d i s − c l i − h r e d i s − m a s t e r R E D I S C L I A U T H = " REDIS_PASSWORD" redis-cli -h redis-master REDISCLI_AUTH=" REDISP​ASSWORD"redis−cli−hredis−masterREDISCLIA​UTH="REDIS_PASSWORD” redis-cli -h redis-replicas

To connect to your database from outside the cluster execute the following commands:

export NODE_IP=$(kubectl get nodes --namespace jwolf -o jsonpath="{.items[0].status.addresses[0].address}")
export NODE_PORT=$(kubectl get --namespace jwolf -o jsonpath="{.spec.ports[0].nodePort}" services redis-master)
REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli -h $NODE_IP -p $NODE_PORT

集群通信必备网络插件

遇到的问题定位:replica无法通过dns或ip访问master导致replica启动失败，之前的k8s集群中的nacos访问不到mysql也是由于集群通信问题，导致通信问题原因是网络插件工作异常（网络插件异常不会影响独立运行的容器——不跟其它容器通信的容器），calica没有ready如下

[root@master k8s]# kubectl get all -A -o wide
NAMESPACE              NAME                                             READY   STATUS    RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
jwolf                  pod/redis-master-0                               1/1     Running   0          6m33s   192.168.166.129   node1    <none>           <none>
jwolf                  pod/redis-replicas-0                             0/1     Running   2          6m33s   192.168.166.128   node1    <none>           <none>
kube-system            pod/calico-kube-controllers-59877c7fb4-2jqnh     1/1     Running   0          5m36s   192.168.219.65    master   <none>           <none>
kube-system            pod/calico-node-66gd7                            0/1     Running   0          3m11s   192.168.1.19      node1    <none>           <none>
kube-system            pod/calico-node-8qp7f                            0/1     Running   0          5m36s   192.168.1.18      master   <none>           <none>
kube-system            pod/etcd-master                                  1/1     Running   11         11d     192.168.1.18      master   <none>           <none>
kube-system            pod/kube-apiserver-master                        1/1     Running   7          8d      192.168.1.18      master   <none>           <none>
kube-system            pod/kube-controller-manager-master               1/1     Running   16         11d     192.168.1.18      master   <none>           <none>
kube-system            pod/kube-proxy-wdgwm                             1/1     Running   0          3m11s   192.168.1.19      node1    <none>           <none>
kube-system            pod/kube-proxy-xfwc7                             1/1     Running   11         11d     192.168.1.18      master   <none>           <none>
kube-system            pod/kube-scheduler-master                        1/1     Running   18         11d     192.168.1.18      master   <none>           <none>
kubernetes-dashboard   pod/dashboard-metrics-scraper-55bc59dffc-lc46j   1/1     Running   8          9d      10.244.219.66     master   <none>           <none>
kubernetes-dashboard   pod/kubernetes-dashboard-5c6dff6c6f-2mzzg        1/1     Running   8          9d      10.244.219.65     master   <none>           <none>

切换为flannel后,coredn自动装上的,所有kube-system组件都ready了，redis replica也连接上master了

[root@master k8s]# kubectl get all -A -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
jwolf         pod/redis-client                     1/1     Running   0          25m   10.244.1.8     node2    <none>           <none>
jwolf         pod/redis-master-0                   1/1     Running   0          64m   10.244.1.7     node2    <none>           <none>
jwolf         pod/redis-replicas-0                 1/1     Running   0          64m   10.244.1.6     node2    <none>           <none>
kube-system   pod/coredns-bccdc95cf-4pd7b          1/1     Running   2          68m   10.244.0.2     master   <none>           <none>
kube-system   pod/coredns-bccdc95cf-nfdd6          1/1     Running   1          68m   10.244.0.3     master   <none>           <none>
kube-system   pod/etcd-master                      1/1     Running   0          67m   192.168.1.18   master   <none>           <none>
kube-system   pod/kube-apiserver-master            1/1     Running   0          64m   192.168.1.18   master   <none>           <none>
kube-system   pod/kube-controller-manager-master   1/1     Running   0          68m   192.168.1.18   master   <none>           <none>
kube-system   pod/kube-flannel-ds-amd64-bdrlb      1/1     Running   0          67m   192.168.1.20   node2    <none>           <none>
kube-system   pod/kube-flannel-ds-amd64-nwdth      1/1     Running   0          68m   192.168.1.18   master   <none>           <none>
kube-system   pod/kube-proxy-cxjr8                 1/1     Running   0          68m   192.168.1.18   master   <none>           <none>
kube-system   pod/kube-proxy-mcsxz                 1/1     Running   0          67m   192.168.1.20   node2    <none>           <none>
kube-system   pod/kube-scheduler-master            1/1     Running   1          68m   192.168.1.18   master   <none>           <none>

flannel.yaml ，下载地址

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
    - configMap
    - secret
    - emptyDir
    - hostPath
  allowedHostPaths:
    - pathPrefix: "/etc/cni/net.d"
    - pathPrefix: "/etc/kube-flannel"
    - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['psp.flannel.unprivileged']
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.12.0-amd64
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.12.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run/flannel
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

欢迎参考作者开源项目

Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes