一、备份脚本

备份脚本k8s-backup.sh内容如下：

#!/bin/bash
# define variable
BACKUP_PATH=/data/$BACKUP_ENV
BACKUP_PATH_DATA=$BACKUP_PATH/yaml/`date +%Y%m%d%H%M%S`
BACKUP_PATH_LOG=$BACKUP_PATH/log
BACKUP_LOG_FILE=$BACKUP_PATH_LOG/k8s-backup-`date +%Y%m%d%H%M%S`.log
# base function
function printlog(){
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1"
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1" >> $BACKUP_LOG_FILE 2>&1 
}
function printlogonly(){
 echo "`date +'%Y-%m-%d %H:%M:%S'` $1" >> $BACKUP_LOG_FILE 2>&1 
}
# set K8s type（此处可根据集群资源自行修改）
CONFIG_TYPE="service deploy ingress configmap secret job cronjob daemonset statefulset"
# make dir
mkdir -p $BACKUP_PATH_DATA
mkdir -p $BACKUP_PATH_LOG
cd $BACKUP_PATH_DATA
# set namespace list
ns_list=`kubectl get ns | awk '{print $1}' | grep -v NAME`
if [ $# -ge 1 ]; then
ns_list="$@"
fi
# define counters
COUNT_NS=0
COUNT_ITEM_IN_NS=0
COUNT_ITEM_IN_TYPE=0
COUNT_ITEM_ALL=0
# print hint
printlog "Backup kubernetes config in namespaces: ${ns_list}"
printlog "Backup kubernetes config for [type: ${CONFIG_TYPE}]."
printlog "If you want to read the record of backup, please input command ' tail -100f ${BACKUP_LOG_FILE} '"
# ask and answer
message="This will backup resources of kubernetes cluster to yaml files."
printlog ${message}
# loop for namespaces
for ns in $ns_list;
do
COUNT_NS=`expr $COUNT_NS + 1`
printlog "Backup No.${COUNT_NS} namespace [namespace: ${ns}]."
COUNT_ITEM_IN_NS=0

## loop for types
for type in $CONFIG_TYPE; 
do
printlogonly "Backup type [namespace: ${ns}, type: ${type}]."
item_list=`kubectl -n $ns get $type | awk '{print $1}' | grep -v NAME | grep -v "No "`
COUNT_ITEM_IN_TYPE=0

## loop for items
for item in $item_list; 
do 
file_name=$BACKUP_PATH_DATA/${ns}_${type}_${item}.yaml
printlogonly "Backup kubernetes config yaml [namespace: ${ns}, type: ${type}, item: ${item}] to file: ${file_name}"
kubectl -n $ns get $type $item -o yaml > $file_name
COUNT_ITEM_IN_NS=`expr $COUNT_ITEM_IN_NS + 1`
COUNT_ITEM_IN_TYPE=`expr $COUNT_ITEM_IN_TYPE + 1`
COUNT_ITEM_ALL=`expr $COUNT_ITEM_ALL + 1`
printlogonly "Backup No.$COUNT_ITEM_ALL file done."
done;

done;
printlogonly "Backup $COUNT_ITEM_IN_TYPE files in [namespace: ${ns}, type: ${type}]."

printlog "Backup ${COUNT_ITEM_IN_NS} files done in [namespace: ${ns}]."
done;

# show stats
printlog "Backup ${COUNT_ITEM_ALL} yaml files in all."
printlog "kubernetes Backup completed, all done."
exit 0

二、集成k8s cronjob

1. 制作备份docker镜像

（1）依赖docker Ubuntu 16.04

docker pull ubuntu:16.04

（2）将k8s环境的/usr/bin/kubectl拷贝进容器中，

docker cp /usr/bin/kubectl xxx:/usr/bin/kubectl

（3）将备份脚本拷贝到容器中，

docker cp /k8s-backup.sh xxx:/k8s-backup.sh
# 并在容器中将新拷入的sh脚本附执行权限
chmod +x k8s-backup.sh

（4）并将对应环境的config文件挂到/root/.kube/config位置（可以先不设置，后续通过k8s configmap进行挂载）,
（5）将新制作的docker镜像提交到镜像仓库，
如：xxx/devops/cron-k8s-backup:2.0

2. 新建k8s config文件对应的configmap

将k8s集群环境中的k8s config文件转换成k8s的configmap，后续可通过k8s将该configmap挂载到对应的cronjob容器中

kubectl create configmap kube-config --from-file=/root/.kube/config -n devops

3. 创建cronjob

注意：
（1）挂载备份目录到nfs
（2）将kube config文件挂载到备份容器中，以便kubectl具有读权限（后续可定制只读权限config文件）
（3）指定备份执行时间（示例为每周五凌晨1点）
（4）启动脚本为./k8s-backup.sh

kind: CronJob
apiVersion: batch/v1beta1
metadata:
  name: devops-backup-k8s
  namespace: devops
  labels:
    name: devops-backup-k8s
    namespace: devops
spec:
  schedule: 0 1 * * 5
  concurrencyPolicy: Forbid
  suspend: false
  jobTemplate:
    metadata:
      creationTimestamp: null
      labels:
        name: devops-backup-k8s
        namespace: devops
    spec:
      parallelism: 1
      completions: 1
      activeDeadlineSeconds: 3600
      backoffLimit: 3
      template:
        metadata:
          creationTimestamp: null
          labels:
            name: devops-backup-k8s
            namespace: devops
        spec:
          volumes:
            - name: backup-dir
              nfs:
                server: 192.168.xx.xx
                path: /data/k8s-backup
            - name: kube-config
              configMap:
                name: kube-config
          containers:
            - name: devops-backup-mysql-ua
              image: 'xxx/devops/cron-k8s-backup:2.0'
              command:
                - ./k8s-backup.sh
              env:
                - name: BACKUP_ENV
                  value: 'fat'
              resources:
                limits:
                  cpu: 500m
                  memory: 512Mi
                requests:
                  cpu: 250m
                  memory: 512Mi
              volumeMounts:
                - name: backup-dir
                  mountPath: /data/
                - name: kube-config
                  mountPath: /root/.kube/
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
              imagePullPolicy: Always
              securityContext:
                privileged: false
                procMount: Default
          restartPolicy: Never
          terminationGracePeriodSeconds: 30
          dnsPolicy: ClusterFirst
          securityContext: {}
          schedulerName: default-scheduler
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3

备份结果