快速指南

以下为快速体验k8s集群的测试、开发环境–单节点部署(aio)，国内环境下比官方的minikube方便、简单很多。

1.基础系统配置

• 准备一台虚机配置内存2G/硬盘30G以上
• 最小化安装Ubuntu 16.04 server或者CentOS 7 Minimal
• 配置基础网络、更新源、SSH登录等

2.下载文件

# 下载工具脚本easzup，举例使用kubeasz版本2.0.2
export release=2.0.2
curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/${release}/easzup
chmod +x ./easzup
# 使用工具脚本下载
./easzup -D

上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录/etc/ansilbe

• /etc/ansible 包含 kubeasz 版本为 ${release} 的发布代码
• /etc/ansible/bin 包含 k8s/etcd/docker/cni 等二进制文件
• /etc/ansible/down 包含集群安装时需要的离线容器镜像
• /etc/ansible/down/packages 包含集群安装时需要的系统基础软件

3.配置 ssh 免密登录

ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa
ssh-copy-id $IP  # $IP 为所有节点地址包括自身，按照提示输入 yes 和 root 密码

4.安装集群

• 4.1 容器化运行 kubeasz，详见文档

./easzup -S

• 4.2 使用默认配置安装 aio 集群

docker exec -it kubeasz easzctl start-aio

5.验证安装

如果提示kubectl: command not found，退出重新ssh登录一下，环境变量生效即可

$ kubectl version                   # 验证集群版本     
$ kubectl get componentstatus       # 验证 scheduler/controller-manager/etcd等组件状态
$ kubectl get node                  # 验证节点就绪 (Ready) 状态
$ kubectl get pod --all-namespaces  # 验证集群pod状态，默认已安装网络插件、coredns、metrics-server等
$ kubectl get svc --all-namespaces  # 验证集群服务状态

• 登录 dashboard可以查看和管理集群，更多内容请查阅dashboard文档

6.清理

以上步骤创建的K8S开发测试环境请尽情折腾，碰到错误尽量通过查看日志、上网搜索、提交issues等方式解决；当然你也可以清理集群后重新创建。

在宿主机上，按照如下步骤清理

• 1.清理集群 docker exec -it kubeasz easzctl destroy 或 docker exec -it kubeasz ansible-playbook /etc/ansible/99.clean.yml
• 2.清理管理节点
  • 清理运行的容器 easzup -C
  • 清理容器镜像 docker system prune -a
  • 停止docker服务 systemctl stop docker
  • 删除下载文件 rm -rf /etc/ansible /etc/docker /opt/kube
  • 删除docker文件

$ umount /var/run/docker/netns/default
$ umount /var/lib/docker/overlay
$ rm -rf /var/lib/docker /var/run/docker

上述清理脚本执行成功后，建议重启节点，以确保清理残留的虚拟网卡、路由等信息。
脚本如下

#!/bin/bash
#--------------------------------------------------
# This script is for: 
# 1. to download the scripts/binaries/images needed for installing a k8s cluster with kubeasz
# 2. to run kubeasz in a container
# @author:   gjmzj
# @usage:    ./easzup
# @repo:     https://github.com/easzlab/kubeasz
# @ref:      https://github.com/kubeasz/dockerfiles
#--------------------------------------------------

set -o nounset
set -o errexit
#set -o xtrace

# default version, can be overridden by cmd line options
export DOCKER_VER=18.09.7
export KUBEASZ_VER=2.0.2
export K8S_BIN_VER=v1.15.0
export EXT_BIN_VER=0.3.0
export SYS_PKG_VER=0.3.2

function install_docker() {
  # check if a container runtime is already installed
  systemctl status docker|grep Active|grep -q running && { echo "[WARN] docker is already running."; return 0; }
  systemctl status containerd|grep Active|grep -q running && { echo "[ERROR] containerd is running, unsupported."; exit 1; }
  
  if [[ "$REGISTRY_MIRROR" == CN ]];then
    DOCKER_URL="https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-${DOCKER_VER}.tgz"
  else
    DOCKER_URL="https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VER}.tgz"
  fi

  mkdir -p /opt/kube/bin /etc/docker /etc/ansible/down
  if [[ -f "/etc/ansible/down/docker-${DOCKER_VER}.tgz" ]];then
    echo "[INFO] docker binaries already existed"
  else
    echo -e "[INFO] \033[33mdownloading docker binaries\033[0m $DOCKER_VER"
    if [[ -e /usr/bin/curl ]];then
      curl -C- -O --retry 3 "$DOCKER_URL" || { echo "[ERROR] downloading docker failed"; exit 1; }
    else
      wget -c "$DOCKER_URL" || { echo "[ERROR] downloading docker failed"; exit 1; }
    fi
    mv ./docker-${DOCKER_VER}.tgz /etc/ansible/down
  fi

  tar zxf /etc/ansible/down/docker-${DOCKER_VER}.tgz -C /etc/ansible/down && \
  mv /etc/ansible/down/docker/* /opt/kube/bin && \
  ln -sf /opt/kube/bin/docker /bin/docker 

  echo "[INFO] generate docker service file"
  cat > /etc/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
Environment="PATH=/opt/kube/bin:/bin:/sbin:/usr/bin:/usr/sbin"
ExecStart=/opt/kube/bin/dockerd
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecReload=/bin/kill -s HUP \$MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

  # configuration for dockerd
  echo "[INFO] generate docker config file"
  if [[ "$REGISTRY_MIRROR" == CN ]];then
    echo "[INFO] prepare register mirror for $REGISTRY_MIRROR"
    cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": [
    "https://dockerhub.azk8s.cn",
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF
  else
    echo "[INFO] standard config without registry mirrors"
    cat > /etc/docker/daemon.json << EOF
{
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF
  fi

  if [[ -e /etc/centos-release || -e /etc/redhat-release ]]; then
    echo "[INFO] turn off selinux in CentOS/Redhat"
    setenforce 0
    echo "SELINUX=disabled" > /etc/selinux/config
  fi

  echo "[INFO] enable and start docker"
  systemctl enable docker
  systemctl daemon-reload && systemctl restart docker && sleep 8
}

function get_kubeasz() {
  # check if kubeasz already existed
  [[ -d "/etc/ansible/roles/kube-node" ]] && { echo "[WARN] kubeasz already existed"; return 0; }

  echo -e "[INFO] \033[33mdownloading kubeasz\033[0m $KUBEASZ_VER"
  echo "[INFO] run a temporary container"
  docker run -d --name temp_easz easzlab/kubeasz:${KUBEASZ_VER} || { echo "[ERROR] download failed."; exit 1; }

  [[ -f "/etc/ansible/down/docker-${DOCKER_VER}.tgz" ]] && mv "/etc/ansible/down/docker-${DOCKER_VER}.tgz" /tmp

  rm -rf /etc/ansible && \
  echo "[INFO] cp kubeasz code from the temporary container" && \
  docker cp temp_easz:/etc/ansible /etc/ansible && \
  echo "[INFO] stop&remove temporary container" && \
  docker rm -f temp_easz

 [[ -f "/tmp/docker-${DOCKER_VER}.tgz" ]] && mv "/tmp/docker-${DOCKER_VER}.tgz" /etc/ansible/down
 return 0
}

function get_k8s_bin() {
  [[ -f "/etc/ansible/bin/kubelet" ]] && { echo "[WARN] kubernetes binaries existed"; return 0; }
  
  echo -e "[INFO] \033[33mdownloading kubernetes\033[0m $K8S_BIN_VER binaries"
  docker pull easzlab/kubeasz-k8s-bin:${K8S_BIN_VER} && \
  echo "[INFO] run a temporary container" && \
  docker run -d --name temp_k8s_bin easzlab/kubeasz-k8s-bin:${K8S_BIN_VER} && \
  echo "[INFO] cp k8s binaries" && \
  docker cp temp_k8s_bin:/k8s /k8s_bin_tmp && \
  mv /k8s_bin_tmp/* /etc/ansible/bin && \
  echo "[INFO] stop&remove temporary container" && \
  docker rm -f temp_k8s_bin && \
  rm -rf /k8s_bin_tmp
}

function get_ext_bin() {
  [[ -f "/etc/ansible/bin/etcdctl" ]] && { echo "[WARN] extral binaries existed"; return 0; }

  echo -e "[INFO] \033[33mdownloading extral binaries\033[0m kubeasz-ext-bin:$EXT_BIN_VER"
  docker pull easzlab/kubeasz-ext-bin:${EXT_BIN_VER} && \
  echo "[INFO] run a temporary container" && \
  docker run -d --name temp_ext_bin easzlab/kubeasz-ext-bin:${EXT_BIN_VER} && \
  echo "[INFO] cp extral binaries" && \
  docker cp temp_ext_bin:/extra /extra_bin_tmp && \
  mv /extra_bin_tmp/* /etc/ansible/bin && \
  echo "[INFO] stop&remove temporary container" && \
  docker rm -f temp_ext_bin && \
  rm -rf /extra_bin_tmp
}

function get_sys_pkg() {
  [[ -f "/etc/ansible/down/packages/chrony_xenial.tar.gz" ]] && { echo "[WARN] system packages existed"; return 0; }

  echo -e "[INFO] \033[33mdownloading system packages\033[0m kubeasz-sys-pkg:$SYS_PKG_VER"
  docker pull easzlab/kubeasz-sys-pkg:${SYS_PKG_VER} && \
  echo "[INFO] run a temporary container" && \
  docker run -d --name temp_sys_pkg easzlab/kubeasz-sys-pkg:${SYS_PKG_VER} && \
  echo "[INFO] cp system packages" && \
  docker cp temp_sys_pkg:/packages /etc/ansible/down && \
  echo "[INFO] stop&remove temporary container" && \
  docker rm -f temp_sys_pkg
}

function get_offline_image() {
  # images needed by k8s cluster
  calicoVer=v3.4.4
  corednsVer=1.5.0
  dashboardVer=v1.10.1
  flannelVer=v0.11.0-amd64
  heapsterVer=v1.5.4
  metricsVer=v0.3.3
  pauseVer=3.1
  traefikVer=v1.7.12

  imageDir=/etc/ansible/down
  [[ -d "$imageDir" ]] || { echo "[ERROR] $imageDir not existed!"; exit 1; }
  
  echo -e "[INFO] \033[33mdownloading offline images\033[0m"

  if [[ ! -f "$imageDir/calico_$calicoVer.tar" ]];then
    docker pull calico/cni:${calicoVer} && \
    docker pull calico/kube-controllers:${calicoVer} && \
    docker pull calico/node:${calicoVer} && \
    docker save -o ${imageDir}/calico_${calicoVer}.tar calico/cni:${calicoVer} calico/kube-controllers:${calicoVer} calico/node:${calicoVer}
  fi
  if [[ ! -f "$imageDir/coredns_$corednsVer.tar" ]];then
    docker pull coredns/coredns:${corednsVer} && \
    docker save -o ${imageDir}/coredns_${corednsVer}.tar coredns/coredns:${corednsVer}
  fi
  if [[ ! -f "$imageDir/dashboard_$dashboardVer.tar" ]];then
    docker pull mirrorgooglecontainers/kubernetes-dashboard-amd64:${dashboardVer} && \
    docker save -o ${imageDir}/dashboard_${dashboardVer}.tar mirrorgooglecontainers/kubernetes-dashboard-amd64:${dashboardVer}
  fi
  if [[ ! -f "$imageDir/flannel_$flannelVer.tar" ]];then
    docker pull easzlab/flannel:${flannelVer} && \
    docker save -o ${imageDir}/flannel_${flannelVer}.tar easzlab/flannel:${flannelVer}
  fi
  if [[ ! -f "$imageDir/heapster_$heapsterVer.tar" ]];then
    docker pull mirrorgooglecontainers/heapster-amd64:${heapsterVer} && \
    docker save -o ${imageDir}/heapster_${heapsterVer}.tar mirrorgooglecontainers/heapster-amd64:${heapsterVer}
  fi
  if [[ ! -f "$imageDir/metrics-server_$metricsVer.tar" ]];then
    docker pull mirrorgooglecontainers/metrics-server-amd64:${metricsVer} && \
    docker save -o ${imageDir}/metrics-server_${metricsVer}.tar mirrorgooglecontainers/metrics-server-amd64:${metricsVer}
  fi
  if [[ ! -f "$imageDir/pause_$pauseVer.tar" ]];then
    docker pull mirrorgooglecontainers/pause-amd64:${pauseVer} && \
    docker save -o ${imageDir}/pause_${pauseVer}.tar mirrorgooglecontainers/pause-amd64:${pauseVer}
  fi
  if [[ ! -f "$imageDir/traefik_$traefikVer.tar" ]];then
    docker pull traefik:${traefikVer} && \
    docker save -o ${imageDir}/traefik_${traefikVer}.tar traefik:${traefikVer}
  fi
  if [[ ! -f "$imageDir/kubeasz_$KUBEASZ_VER.tar" ]];then
    docker pull easzlab/kubeasz:${KUBEASZ_VER} && \
    docker save -o ${imageDir}/kubeasz_${KUBEASZ_VER}.tar easzlab/kubeasz:${KUBEASZ_VER}
  fi
}

function download_all() {
  install_docker && \
  get_kubeasz && \
  get_k8s_bin && \
  get_ext_bin && \
  get_sys_pkg && \
  get_offline_image
}

function start_kubeasz_docker() {
  [[ -d "/etc/ansible/roles/kube-node" ]] || { echo "[ERROR] not initialized. try 'easzup -D' first."; exit 1; }

  # get host's IP
  host_if=$(ip route|grep default|cut -d' ' -f5)
  host_ip=$(ip a|grep "$host_if$"|awk '{print $2}'|cut -d'/' -f1)
  echo "[INFO] get host IP: $host_ip"

  # allow ssh login using key locally
  if [[ ! -e /root/.ssh/id_rsa ]]; then
    echo "[INFO] generate ssh key pair"
    ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa > /dev/null
    cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
    ssh-keyscan -t ecdsa -H "$host_ip" >> /root/.ssh/known_hosts
  fi

  # create a link '/usr/bin/python' in Ubuntu1604
  if [[ ! -e /usr/bin/python && -e /etc/debian_version ]]; then
    echo "[INFO] create a soft link '/usr/bin/python'"
    ln -s /usr/bin/python3 /usr/bin/python
  fi

  # 
  docker load -i /etc/ansible/down/kubeasz_${KUBEASZ_VER}.tar

  # run kubeasz docker container
  echo "[INFO] run kubeasz in a container"
  docker run --detach \
      --name kubeasz \
      --restart always \
      --env HOST_IP="$host_ip" \
      --volume /etc/ansible:/etc/ansible \
      --volume /root/.kube:/root/.kube \
      --volume /root/.ssh/id_rsa:/root/.ssh/id_rsa:ro \
      --volume /root/.ssh/id_rsa.pub:/root/.ssh/id_rsa.pub:ro \
      --volume /root/.ssh/known_hosts:/root/.ssh/known_hosts:ro \
      easzlab/kubeasz:${KUBEASZ_VER} sleep 36000
}

function clean_container() {
 echo "[INFO] clean all running containers"
 docker ps -a|awk 'NR>1{print $1}'|xargs docker rm -f 
} 

function usage() {
  cat <<EOF
Usage: easzup [options] [args]
  option: -{DdekSz}
    -C         stop&clean all local containers
    -D         download all into /etc/ansible
    -S         start kubeasz in a container
    -d <ver>   set docker-ce version, default "$DOCKER_VER"
    -e <ver>   set kubeasz-ext-bin version, default "$EXT_BIN_VER"
    -k <ver>   set kubeasz-k8s-bin version, default "$K8S_BIN_VER"
    -m <str>   set docker registry mirrors, default "CN"(used in Mainland,China)
    -p <ver>   set kubeasz-sys-pkg version, default "$SYS_PKG_VER"
    -z <ver>   set kubeasz version, default "$KUBEASZ_VER"
  
see more at https://github.com/kubeasz/dockerfiles
EOF
} 

### Main Lines ##################################################
function main() {
  # check if use bash shell
  readlink /proc/$$/exe|grep -q "dash" && { echo "[ERROR] you should use bash shell, not sh"; exit 1; }
  # check if use with root
  [[ "$EUID" -ne 0 ]] && { echo "[ERROR] you should run this script as root"; exit 1; }
  
  [[ "$#" -eq 0 ]] && { usage >&2; exit 1; }
  
  export REGISTRY_MIRROR="CN"
  ACTION=""
  while getopts "CDSd:e:k:m:p:z:" OPTION; do
      case "$OPTION" in
        C)
          ACTION="clean_container"
          ;;
        D)
          ACTION="download_all"
          ;;
        S)
          ACTION="start_kubeasz_docker"
          ;;
        d)
          export DOCKER_VER="$OPTARG"
          ;;
        e)
          export EXT_BIN_VER="$OPTARG"
          ;;
        k)
          export K8S_BIN_VER="$OPTARG"
          ;;
        m)
          export REGISTRY_MIRROR="$OPTARG"
          ;;
        p)
          export SYS_PKG_VER="$OPTARG"
          ;;
        z)
          export KUBEASZ_VER="$OPTARG"
          ;;
        ?)
          usage
          exit 1
          ;;
      esac
  done
  
  [[ "$ACTION" == "" ]] && { echo "[ERROR] illegal option"; usage; exit 1; }
  
  # excute cmd "$ACTION" 
  echo -e "[INFO] \033[33mAction begin\033[0m : $ACTION"
  ${ACTION} || { echo -e "[ERROR] \033[31mAction failed\033[0m : $ACTION"; return 1; }
  echo -e "[INFO] \033[32mAction successed\033[0m : $ACTION"
}

main "$@"