使用Ansible + kubeadm部署K8S集群

纯属练手，这几天看了Ansible相关的资料，练习一下，加深了解！

1、环境清单

1.1、系统清单

IP	Role	OS
192.168.119.160	k8s-master	CentOS 7
192.168.119.161	K8s-node-1	CentOS 7
192.168.119.162	K8s-node-2	CentOS 7
192.168.119.169	ansible-manage	CentOS 7

1.2、软件清单

1.2.1、Ansible

[chenlei@node-ansible ~]$ ansible --version
ansible 2.5.0 (devel d1f19125a5) last updated 2018/02/08 21:21:11 (GMT +800)
  config file = None
  configured module search path = [u'/home/chenlei/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/chenlei/ansible/lib/ansible
  executable location = /home/chenlei/ansible/bin/ansible
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

1.2.2、Kubernetes

参见上一篇文章内容，本文是上一篇kubeadm的ansible版本。

1.3、镜像清单

参见上一篇文章内容，本文是上一篇kubeadm的ansible版本。

2、部署说明

2.1、安装Ansible

### 安装git ###
[chenlei@node-ansible ~]$ sudo yum install -y git

### 从GitHub上下载源代码 ###
[chenlei@node-ansible ~]$ git clone https://github.com/ansible/ansible.git --recursive
正克隆到 'ansible'...
remote: Counting objects: 302399, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 302399 (delta 11), reused 9 (delta 0), pack-reused 302352
接收对象中: 100% (302399/302399), 109.50 MiB | 16.00 KiB/s, done.
处理 delta 中: 100% (190336/190336), done.
[chenlei@node-ansible ~]$ ll
drwxrwxr-x. 14 chenlei chenlei 4096 2月  28 10:48 ansible

### 安装 ###
[chenlei@node-ansible ~]$ source ansible/hacking/env-setup
Ansible now needs setuptools in order to build. Install it using your package manager (usually python-setuptools) or via pip (pip install setuptools).

Setting up Ansible to run out of checkout...

PATH=/home/chenlei/ansible/bin:/home/chenlei/ansible/test/runner:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/chenlei/.local/bin:/home/chenlei/bin
PYTHONPATH=/home/chenlei/ansible/lib:
MANPATH=/home/chenlei/ansible/docs/man:

Remember, you may wish to specify your host file with -i

Done!

这里直接以源码方式运行Ansible、从github上下载源码之后，根据源码内的脚本设置好环境变量即可

这里的安装只是设置环境变量，系统重启后即失效

2.2、安装setuptools

### 下载setuptools ###
[chenlei@node-ansible ~]$ wget https://pypi.python.org/packages/6c/54/f7e9cea6897636a04e74c3954f0d8335cc38f7d01e27eec98026b049a300/setuptools-38.5.1.zip#md5=1705ae74b04d1637f604c336bb565720
--2018-02-28 11:01:29--  https://pypi.python.org/packages/6c/54/f7e9cea6897636a04e74c3954f0d8335cc38f7d01e27eec98026b049a300/setuptools-38.5.1.zip
正在解析主机 pypi.python.org (pypi.python.org)... 151.101.228.223, 2a04:4e42:36::223
正在连接 pypi.python.org (pypi.python.org)|151.101.228.223|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：737995 (721K) [binary/octet-stream]
正在保存至: “setuptools-38.5.1.zip”

100%[====================================================================>] 737,995      378KB/s 用时 1.9s   

2018-02-28 11:01:32 (378 KB/s) - 已保存 “setuptools-38.5.1.zip” [737995/737995])

[chenlei@node-ansible ~]$ ll
drwxrwxr-x. 14 chenlei chenlei   4096 2月  28 10:48 ansible
-rw-rw-r--.  1 chenlei chenlei 737995 2月   6 18:59 setuptools-38.5.1.zip

### 安装setuptools ###
[chenlei@node-ansible ~]$ unzip setuptools-38.5.1.zip
[chenlei@node-ansible ~]$ cd setuptools-38.5.1
[chenlei@node-ansible setuptools-38.5.1]$ sudo python setup.py install

### 安装pip ###
[chenlei@node-ansible ~]$ sudo easy_install pip

setuptools即easy_install，用于安装pip，安装pip是为了安装ansible的依赖

2.3、安装ansible的依赖

### 安装Jinja2 ###
[chenlei@node-ansible ~]$ sudo pip install Jinja2
### 安装PyYAML ###
[chenlei@node-ansible ~]$ sudo pip install PyYAML

2.4、测试ansible

1、在ansible-master上生成RSA密钥对，用于在SSH链接时免密登录

### 生成RSA密钥对 ###
[chenlei@node-ansible ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/chenlei/.ssh/id_rsa): 
Created directory '/home/chenlei/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/chenlei/.ssh/id_rsa.
Your public key has been saved in /home/chenlei/.ssh/id_rsa.pub.
The key fingerprint is:
b1:ed:58:8f:81:1f:de:6f:54:35:43:bb:36:d0:fa:73 chenlei@node-ansible
The key's randomart image is:
+--[ RSA 2048]----+
|              .. |
|              .oo|
|        .    . o+|
|         =    o o|
|        S =  . = |
|         * *  + .|
|        . = o. oE|
|             .. o|
|             ..  |
+-----------------+

### 将公钥复制到其他被管节点上 ###
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.160
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.161
[chenlei@node-ansible ~]$ ssh-copy-id root@192.168.119.162

### 测试ssh连接是否需要密码 ###
[chenlei@node-ansible ~]$ ssh root@192.168.119.160
[chenlei@node-ansible ~]$ ssh root@192.168.119.161
[chenlei@node-ansible ~]$ ssh root@192.168.119.162

2、创建ansible默认hosts文件

[chenlei@node-ansible ~]$ vi /etc/ansible/hosts
192.168.119.160
192.168.119.161
192.168.119.162

### 测试 ###
[chenlei@node-ansible ~]$ ansible all -m ping -u root
192.168.119.160 | SUCCESS => {
    "changed": false, 
    "failed": false, 
    "ping": "pong"
}
192.168.119.162 | SUCCESS => {
    "changed": false, 
    "failed": false, 
    "ping": "pong"
}
192.168.119.161 | SUCCESS => {
    "changed": false, 
    "failed": false, 
    "ping": "pong"
}

all表示全部主机，-m用来指定使用的module，-u用来指定远端用户

2.5、编写playbook

2.5.1、目录结构

[chenlei@node-ansible test-01]$ pwd
/home/chenlei/playbooks/test-01
[chenlei@node-ansible test-01]$ tree .
.
├── application.yml             # 应用部署playbook文件
├── group_vars/                 # group变量存放路径
│   └── master                  # master组变量
├── host_vars/                  # host变量存放目录，目前没有使用
├── kubernetes.yml              # 集群部署playbook文件
├── master.yml                  # 部署master的playbook文件
├── node.yml                    # 部署node的playbook文件
├── production                  # 生产环境的inventory文件，目前没有使用
├── reset.yml                   # 集群重置的playbook文件
├── roles/                      # 存放role的文件夹，role本质上就是一个固定目录结构
│   ├── common/                 # common role，包含公共预处理task和文件
│   │   ├── files/
│   │   │   └── k8s.conf
│   │   └── tasks/
│   │       └── main.yml
│   ├── dashboard/              # dashboard role，包含dashboard部署task
│   │   └── tasks/
│   │       └── main.yml
│   ├── docker/                 # docker role，包括docker安装启动task和镜像导入task
│   │   ├── defaults/
│   │   ├── files/
│   │   ├── handlers/
│   │   ├── meta/
│   │   ├── tasks/
│   │   │   └── main.yml
│   │   ├── templates/
│   │   └── vars/
│   ├── kubelet/                # kubelet role，包含kubelet安装启动task
│   │   └── tasks/
│   │       └── main.yml
│   ├── master/                 # master role，包含master初始化和设置集群加入脚本fact
│   │   └── tasks/
│   │       └── main.yml
│   └── node/                   # node role，包含node加入集群的task
│       └── tasks/
│           └── main.yml
└── stage                       # 演示环境的inventory文件

22 directories, 15 files

为了学习和了解ansible，上述很多目录列举出来，但是没有使用，可以删除。

2.5.2、文件内容

1、主playbook文件：kubernetes.yml

[chenlei@node-ansible test-01]$ vi kubernetes.yml
---
- import_playbook: master.yml
- import_playbook: node.yml
- import_playbook: application.yml

[chenlei@node-ansible test-01]$ vi master.yml
---
- hosts: master
  roles:
  - common
  - docker
  - kubelet
  - master

[chenlei@node-ansible test-01]$ vi node.yml
---

- hosts: node
  roles:
  - common
  - docker
  - kubelet
  - node

[chenlei@node-ansible test-01]$ vi application.yml
---
- hosts: master
  roles:
  - dashboard

Ansible使用这种文件拆分和引用的方式来更好的重用和管理配置信息

2、集群重置playbook文件：reset.yml

[chenlei@node-ansible test-01]$ vi reset.yml
---

- hosts: master,node
  tasks:
  - name: Reset node
    shell: kubeadm reset

3、Inventory文件内容

[chenlei@node-ansible test-01]$ vi stage 
[master]
192.168.119.160

[node]
192.168.119.161
192.168.119.162

本文使用stage清单文件，production文件为空。标准情况下，stage用于配置演示或测试环境，production存放生产环境。

中括号表示group

4、角色配置文件

• common role

[chenlei@node-ansible roles]$ vi common/files/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

[chenlei@node-ansible roles]$ vi common/tasks/main.yml
---

- name: Be sure swap is off
  command: /usr/sbin/swapoff -a

- name: Be sure selinux is off
  command: /usr/sbin/setenforce 0

- name: Stop and disable firewall
  systemd:
    name: firewalld
    state: stopped
    enabled: no

- name: Set sysctl file
  copy:
    src: k8s.conf
    dest: /etc/sysctl.d/k8s.conf

- name: Make up sysctl file
  shell: sysctl --system

• docker role

[chenlei@node-ansible roles]$ vi docker/tasks/main.yml
---

- name: Be sure docker is installed
  yum:
    pkg: docker
    state: installed
  tags: docker

- name: Be sure docker is running and enable
  systemd:
    name: docker
    enabled: yes
    state: started
    daemon_reload: yes

- name: Send docker images
  copy: src=~/kubernetes/images/ dest=~/kubernetes/images/

- name: Load docker images
  shell: for file in ~/kubernetes/images/*; do docker load -i $file; done;

• kubelet role

[chenlei@node-ansible roles]$ vi kubelet/tasks/main.yml
---

- name: Send rpm packages
  copy: src=~/kubernetes/rpm/ dest=~/kubernetes/rpm/

- name: Be sure kubelet is installed
  yum:
    name: /root/kubernetes/rpm/kubeadm-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubectl-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubelet-1.9.2-0.x86_64.rpm,/root/kubernetes/rpm/kubernetes-cni-0.6.0-0.x86_64.rpm,/root/kubernetes/rpm/socat-1.7.3.2-2.el7.x86_64.rpm
    state: present

- name: Be sure kubelet is enabled
  systemd:
    name: kubelet
    enabled: yes

• master role

[chenlei@node-ansible roles]$ vi master/tasks/main.yml
---

- name: Exec kubeadm init
  shell: kubeadm init --kubernetes-version=v1.9.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address={{ apiserver_address }}

- name: Set cluster config
  shell: HOME=~; sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config

- name: Generate cluster join script
  shell: kubeadm token create --print-join-command
  register: cluster_join

- set_fact: cluster_join_script={{ cluster_join.stdout }}

- name: send yaml file
  copy:
    src=~/kubernetes/yml/
    dest=~/kubernetes/yml/

- name: set up cni
  shell: kubectl apply -f ~/kubernetes/yml/kube-flannel.yml

这里设置了cluster_join_script，用于在node中执行kubeadm join

• node role

[chenlei@node-ansible roles]$ vi node/tasks/main.yml
---

- set_fact: cluster_join_script={{ hostvars[groups['master'][0]]['cluster_join_script'] }}

- name: Join kubernetes cluster
  shell: echo '' | {{ cluster_join_script }}
  when: cluster_join_script is defined

cluster_join_script取自master组中的第一个主机的cluster_join_script，也就是上面设置的内容。hostvars是ansible内置变量。

• dashboard role

[chenlei@node-ansible roles]$ vi dashboard/tasks/main.yml
---

- name: Deploy kubernetes dashboard
  shell: kubectl apply -f ~/kubernetes/yml/kubernetes-dashboard.yml

- name: Create dashboard account
  shell: kubectl apply -f ~/kubernetes/yml/kubernetes-dashboard-rbac-admin.yml

2.6、运行playbook

[chenlei@node-ansible test-01]$ ansible-playbook -i stage -u root kubernetes.yml

### 重置 ###
[chenlei@node-ansible test-01]$ ansible-playbook -i stage -u root reset.yml

2.7、检查k8s集群状态

[root@k8s-master ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"} 

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
k8s-master   Ready     master    4m        v1.9.2
k8s-node-1   Ready     <none>    3m        v1.9.2
k8s-node-2   Ready     <none>    3m        v1.9.2

[root@k8s-master ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE       IP                NODE
kube-system   etcd-k8s-master                         1/1       Running   0          2m        192.168.119.160   k8s-master
kube-system   kube-apiserver-k8s-master               1/1       Running   0          2m        192.168.119.160   k8s-master
kube-system   kube-controller-manager-k8s-master      1/1       Running   0          2m        192.168.119.160   k8s-master
kube-system   kube-dns-6f4fd4bdf-mwddx                3/3       Running   0          3m        10.244.0.19       k8s-master
kube-system   kube-flannel-ds-nwxcl                   1/1       Running   0          2m        192.168.119.162   k8s-node-2
kube-system   kube-flannel-ds-pdxhs                   1/1       Running   0          2m        192.168.119.161   k8s-node-1
kube-system   kube-flannel-ds-qrkwd                   1/1       Running   0          3m        192.168.119.160   k8s-master
kube-system   kube-proxy-7lsqw                        1/1       Running   0          2m        192.168.119.161   k8s-node-1
kube-system   kube-proxy-f9kps                        1/1       Running   0          2m        192.168.119.162   k8s-node-2
kube-system   kube-proxy-g47nx                        1/1       Running   0          3m        192.168.119.160   k8s-master
kube-system   kube-scheduler-k8s-master               1/1       Running   0          2m        192.168.119.160   k8s-master
kube-system   kubernetes-dashboard-845747bdd4-gcj47   1/1       Running   0          2m        10.244.0.20       k8s-master

[root@k8s-master ~]# kubectl get svc --all-namespaces
NAMESPACE     NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       kubernetes             ClusterIP   10.96.0.1        <none>        443/TCP         3m
kube-system   kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   3m
kube-system   kubernetes-dashboard   NodePort    10.111.158.221   <none>        443:32286/TCP   2m

2.8、访问dashboard

浏览器访问：https://192.168.119.160:32286

3、参考资料

http://www.ansible.com.cn/index.html

http://docs.ansible.com/ansible/latest/index.html

官方文档的Module index中有大量的使用范例可供参考