K8S(二)安装配置篇
架构图。
·
架构图
Kubernetes节点
在这张系统架构图中，我们把服务分为运行在工作节点上的服务和组成集群级别控制板的服务。
Kubernetes节点有运行应用容器必备的服务，而这些都是受Master的控制。
每次个节点上当然都要运行Docker。Docker来负责所有具体的映像下载和容器运行。
Kubernetes主要由以下几个核心组件组成：
- etcd保存了整个集群的状态；
- apiserver提供了资源操作的唯一入口，并提供认证、授权、访问控制、API注册和发现等机制；
- controller manager负责维护集群的状态，比如故障检测、自动扩展、滚动更新等；
- scheduler负责资源的调度，按照预定的调度策略将Pod调度到相应的机器上；
- kubelet负责维护容器的生命周期，同时也负责Volume（CVI）和网络（CNI）的管理；
- Container runtime负责镜像管理以及Pod和容器的真正运行（CRI）；
- kube-proxy负责为Service提供cluster内部的服务发现和负载均衡；
除了核心组件，还有一些推荐的Add-ons：
- kube-dns负责为整个集群提供DNS服务
- Ingress Controller为服务提供外网入口
- Heapster提供资源监控
- Dashboard提供GUI
- Federation提供跨可用区的集群
- Fluentd-elasticsearch提供集群日志采集、存储与查询
参考学习文档 https://github.com/easzlab/kubeasz
| Host Name | Role | IP | conf | version |
|---|---|---|---|---|
| master-bak | master (用于测试添加) | 10.2.33.101 | 4C8G | centos7.5 |
| master-1 | master etcd 部署节点 | 10.2.33.100 | 4C8G | centos7.5 |
| master-2 | master etcd | 10.2.33.99 | 4C8G | centos7.5 |
| master-3 | master etcd | 10.2.33.98 | 4C8G | centos7.5 |
| node-1 | node | 10.2.33.97 | 8C16G | centos7.5 |
| node-2 | node | 10.2.33.96 | 8C16G | centos7.5 |
| node-3 | node | 10.2.33.95 | 8C16G | centos7.5 |
| node-bak | node(用于测试添加) | 10.2.33.94 | 8C16G | centos7.5 |
所有服务器初始化、安装python2.7
部署节点配置如下：
yum -y install ansible
ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa root@10.2.33.95-100
curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/3.1.1/ezdown
chmod +x ezdown
./ezdown -D
上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录/etc/kubeasz
cd /etc/kubeasz/
./ezctl new k8s-01
cd clusters/k8s-01/
然后根据提示配置'/etc/kubeasz/clusters/k8s-01/hosts' 和 '/etc/kubeasz/clusters/k8s-01/config.yml'：
根据前面节点规划修改hosts 文件和其他集群层面的主要配置选项；其他集群组件等配置项可以在config.yml 文件中修改。
# 一键安装
ezctl setup k8s-01 all
# 或者分步安装，具体使用 ezctl help setup 查看分步安装帮助信息
# ezctl setup k8s-01 01
# ezctl setup k8s-01 02
# ezctl setup k8s-01 03
# ezctl setup k8s-01 04
# ezctl setup k8s-01 05
# ezctl setup k8s-01 06
# ezctl setup k8s-01 07##########dashboard#########
[root@localhost ssl]# kubectl get svc -n kube-system|grep dashboard
dashboard-metrics-scraper ClusterIP 10.68.37.90 <none> 8000/TCP 16h
kubernetes-dashboard NodePort 10.68.154.239 <none> 443:32543/TCP 16h
[root@localhost ssl]# kubectl get svc -n kube-system -o wide |grep dashboard
dashboard-metrics-scraper ClusterIP 10.68.37.90 <none> 8000/TCP 16h k8s-app=dashboard-metrics-scraper
kubernetes-dashboard NodePort 10.68.154.239 <none> 443:32543/TCP 16h k8s-app=kubernetes-dashboard
#kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Name: admin-user-token-bhpgw
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: e14095d4-098b-42d3-8e04-84ac8e47e5e6
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1350 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6InA1OFZxRzlrZUVrc0c0OTRlZ0pKMWFDVnhhMWxhYVRJYnRWNXZsQU03NTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWJocGd3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlMTQwOTVkNC0wOThiLTQyZDMtOGUwNC04NGFjOGU0N2U1ZTYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.bp-l1M-yu4qnpAyeCuVoptlW1g8awQDI0XD89dr-RCNDQGpnRVpt5Ws5K0yLRSilA8Q4YdlHHQMckGmxdom0TiFk9Vr5BukdO51g7VIyOd__ZnIbRRRTtuNpO7PvkxzxyVRW9a5I89JDLPb2ptKLHBrolVGJK3DHZJbqc29ATYlYxoN2cDNYmNjacGXIyKfAxZkrcxxMEx9AebLAfbGQaVzKXPUapPMhTDo-flcR9Z3xBKXXoS0AQ3pA8rGOGi7qmsvXYEYvW_RKj2zCSZw7Cr_5Qnb6_EKB6lmSRGa7Uec5TeMyiw-FYSQmaVVOOF1wk-bDxauaJmh7prqRNw85tA
火狐浏览器访问
https://master_ip:32543 (输入以上的token,admin管理用户登陆)##########Prometheus#########
/etc/kubeasz/clusters/k8s-01/config.yml 中配置项 prom_install: "yes"
ezctl setup k8s-01 07 (多执行几次，会下载失败，多等会安装慢)
#查询安装状态
[root@localhost k8s-01]# kubectl get pod,svc -n monitor
NAME READY STATUS RESTARTS AGE
pod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 63m
pod/prometheus-grafana-55c5f574d9-z7fpk 2/2 Running 0 64m
pod/prometheus-kube-prometheus-operator-5f6774b747-8qvrx 1/1 Running 0 64m
pod/prometheus-kube-state-metrics-5f89586745-4zqqs 1/1 Running 0 64m
pod/prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 1 (23m ago) 63m
pod/prometheus-prometheus-node-exporter-2smhh 1/1 Running 0 64m
pod/prometheus-prometheus-node-exporter-k24jp 1/1 Running 0 64m
pod/prometheus-prometheus-node-exporter-n5wxb 1/1 Running 0 64m
pod/prometheus-prometheus-node-exporter-vm5zl 1/1 Running 0 64m
pod/prometheus-prometheus-node-exporter-xwpv4 1/1 Running 0 64m
pod/prometheus-prometheus-node-exporter-zjv95 1/1 Running 0 64m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 63m
service/prometheus-grafana NodePort 10.68.202.166 <none> 80:30903/TCP 64m
service/prometheus-kube-prometheus-alertmanager NodePort 10.68.152.180 <none> 9093:30902/TCP 64m
service/prometheus-kube-prometheus-operator NodePort 10.68.58.144 <none> 443:30900/TCP 64m
service/prometheus-kube-prometheus-prometheus NodePort 10.68.236.38 <none> 9090:30901/TCP 64m
service/prometheus-kube-state-metrics ClusterIP 10.68.100.203 <none> 8080/TCP 64m
service/prometheus-operated ClusterIP None <none> 9090/TCP 63m
service/prometheus-prometheus-node-exporter ClusterIP 10.68.201.76 <none> 9100/TCP 64m
访问prometheus的web界面：http://$NodeIP:30901
访问alertmanager的web界面：http://$NodeIP:30902
访问grafana的web界面：http://$NodeIP:30903 (默认用户密码 admin:Admin1234! 密码在下面文件中存储)
/etc/kubeasz/roles/cluster-addon/templates/prometheus/values.yaml.j2##########ingress#########
cd /etc/kubeasz/manifests/ingress/nginx-ingress
[root@master-01 nginx-ingress]# cat nginx-ingress.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
hostNetwork: true
nodeSelector:
nginx: "nginx"
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
#image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
#使用以下镜像，方便国内下载加速
image: jmgao1983/nginx-ingress-controller:0.21.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
# hostPort可以直接使用node节点的网络端口暴露服务
#- name: mysql
# containerPort: 3306
# hostPort: 3306
#- name: dns
# containerPort: 53
# hostPort: 53
# protocol: UDP
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
[root@localhost nginx-ingress]# kubectl create -f nginx-ingress.yaml
[root@master-01 nginx-ingress]# cat nginx-ingress-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
# 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围
# 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问
nodePort: 23456
- name: https
port: 443
targetPort: 443
protocol: TCP
# 集群hosts文件中设置的 NODE_PORT_RANGE 作为 NodePort的可用范围
# 从默认20000~40000之间选一个可用端口，让ingress-controller暴露https
nodePort: 23457
- name: test-mysql
port: 3306
targetPort: 3306
protocol: TCP
nodePort: 23306
- name: test-mysql-read
port: 3307
targetPort: 3307
protocol: TCP
nodePort: 23307
- name: test-dns
port: 53
targetPort: 53
protocol: UDP
nodePort: 20053
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
[root@localhost nginx-ingress]# kubectl create -f nginx-ingress-svc.yaml
[root@master-01 nginx-ingress]# cat tcp-services-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
3306: "mariadb/mydb-mariadb:3306"
3307: "mariadb/mydb-mariadb-slave:3306"
9200: "elastic/elasticsearch:9200"
5044: "elastic/logstash:5044"
5601: "elastic/kibana:5601"
[root@localhost nginx-ingress]# kubectl create -f tcp-services-configmap.yaml
[root@localhost nginx-ingress]# kubectl get namespace
NAME STATUS AGE
default Active 22h
ingress-nginx Active 20h
kube-node-lease Active 22h
kube-public Active 22h
kube-system Active 22h
test-tengine Active 20h
[root@localhost nginx-ingress]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/nginx-ingress-controller-6f4c78bb57-fshh7 1/1 Running 0 20h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx NodePort 10.68.20.223 <none> 80:23456/TCP,443:23457/TCP 20h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx-ingress-controller 1/1 1 1 20h
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-ingress-controller-6f4c78bb57 1 1 1 20h
###以后只需要更新 nginx-ingress-svc.yaml tcp-services-configmap.yaml 即可
##########node操作#########
#新增加节点10.2.33.94
#新增kube_node节点大致流程为：(参考ezctl 里面add-node函数 和 /etc/kubeasz/playbooks/22.addnode.yml)
首先配置ssh免密码登录新增节点 ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.2.33.94
执行添加 [root@localhost kubeasz]# ./ezctl add-node k8s-01 10.2.33.94 (添加非标准口 + ansible_ssh_port=10022)
[root@localhost kubeasz]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.2.33.100 Ready,SchedulingDisabled master 23h v1.21.0
10.2.33.94 Ready node 10m v1.21.0
10.2.33.95 Ready node 23h v1.21.0
10.2.33.96 Ready node 23h v1.21.0
10.2.33.97 Ready node 23h v1.21.0
10.2.33.98 Ready,SchedulingDisabled master 23h v1.21.0
10.2.33.99 Ready,SchedulingDisabled master 23h v1.21.0
删除node节点需执行 [root@localhost kubeasz]# ./ezctl del-node k8s-01 10.2.33.94
##########master操作#########
#新增加节点10.2.33.101
新增kube_master节点大致流程为：(参考ezctl 中add-master函数和/etc/kubeasz/playbooks/23.addmaster.yml)
首先配置ssh免密码登录新增节点 ssh-copy-id -i /root/.ssh/id_rsa.pub root@10.2.33.101
执行添加 [root@localhost kubeasz]# ./ezctl add-master k8s-01 10.2.33.101
#在新节点master查看服务状态
$ systemctl status kube-apiserver
$ systemctl status kube-controller-manager
$ systemctl status kube-scheduler
# 查看集群节点，可以看到新 master节点 Ready, 并且禁止了POD 调度功能
[root@localhost kubeasz]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.2.33.100 Ready,SchedulingDisabled master 23h v1.21.0
10.2.33.101 Ready,SchedulingDisabled master 3m57s v1.21.0 # 新增 master节点
10.2.33.94 Ready node 22m v1.21.0
10.2.33.95 Ready node 23h v1.21.0
10.2.33.96 Ready node 23h v1.21.0
10.2.33.97 Ready node 23h v1.21.0
10.2.33.98 Ready,SchedulingDisabled master 23h v1.21.0
10.2.33.99 Ready,SchedulingDisabled master 23h v1.21.0
删除master节点执行 [root@localhost kubeasz]# ./ezctl del-master k8s-01 10.2.33.101
##########etcd操作#########
#新增加节点(注etcd服务器必须是奇数)
首先确认配置 ssh 免密码登录，然后执行 (假设待操作节点为 192.168.1.11，集群名称k8s-01)：
增加 etcd 节点：$ ezctl add-etcd k8s-01 192.168.1.11
删除 etcd 节点：$ ezctl del-etcd k8s-01 192.168.1.11
#########k8s#使用#gfs#########
gfs安装略https://blog.csdn.net/zhangxueleishamo/article/details/103600011?spm=1001.2014.3001.5502
#node上安装gfs客户端
yum -y install glusterfs glusterfs-fuse
#查看gfs服务器的端口 netstat -ntlp
#新增gfs在相关namespace的Endpoints
[root@localhost nfs]# cat k8snfs.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: k8snfs
namespace: test-tengine
subsets:
- addresses:
- ip: 10.2.33.40
- ip: 10.2.33.41
ports:
- port: 49154
protocol: TCP
[root@localhost nfs]# kubectl create -f k8snfs.yaml
[root@localhost nfs]# kubectl get EndPoints -n test-tengine
NAME ENDPOINTS AGE
k8snfs 10.2.33.40:49154,10.2.33.41:49154 17m
创建pod挂载endpoints
[root@k8s-master01 K8s_conf]# cat Deployment_tengine_online.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: online-tengine
namespace: test-tengine
spec:
selector:
matchLabels:
app: online-tengine-labels
minReadySeconds: 1
progressDeadlineSeconds: 60
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
replicas: 2
template:
metadata:
name: online-tengine
labels:
app: online-tengine-labels
spec:
containers:
- name: online-tengine
image: registry.yoyi.tv/test-tengine/online-tengine:21
ports:
- containerPort: 8107
command: ['sh']
args:
- "-c"
- |
mkdir -p /home/mount/$HOSTNAME
rm -rf /data/server/tengine/logs
ln -s /home/mount/$HOSTNAME /data/server/tengine/logs
mkfifo /data/server/tengine/logs/access.log
nohup cat /data/server/tengine/logs/access.log | nohup /usr/sbin/cronolog /data/server/tengine/logs/access_%y%m%d%h.log &
ping -c 20 127.0.0.1
/usr/bin/supervisord -n -c /etc/supervisord.conf
volumeMounts:
- name: k8snfs
mountPath: "/data/shared_data"
- name: plat-time
mountPath: "/etc/localtime"
- name: plat-time-zone
mountPath: "/etc/timezore"
volumes:
- name: k8snfs
glusterfs:
endpoints: k8snfs
path: k8snfs
readOnly: false
- name: plat-time
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
- name: plat-time-zone
hostPath:
path: /etc/timezone
[root@k8s-master01 K8s_conf]# cat Server_tengine_online.yaml
apiVersion: v1
kind: Service
metadata:
name: online-tengine-svc
namespace: test-tengine
spec:
type: NodePort
ports:
- port: 8107
targetPort: 8109
selector:
app: online-tengine-labels##########Elasticsearch集群#########
在生产环境下，Elasticsearch 集群由不同的角色节点组成：
master 节点：参与主节点选举，不存储数据；建议3个以上，维护整个集群的稳定可靠状态
data 节点：不参与选主，负责存储数据；主要消耗磁盘，内存
client 节点：不参与选主，不存储数据；负责处理用户请求，实现请求转发，负载均衡等功能
用helm chart来部署 (https://github.com/helm/charts/tree/master/incubator/elasticsearch)
1.安装 helm: 以本项目安全安装helm（https://github.com/easzlab/kubeasz/blob/master/docs/guide/helm.md）为例
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz (如下载慢，请用迅雷 https://github.com/helm/helm/releases)
tar -zxvf helm-v3.2.1-linux-amd64.tar.gz
mv ./linux-amd64/helm /usr/bin
helm repo add stable http://mirror.azure.cn/kubernetes/charts/ 2.准备 PV: 以本项目K8S 集群存储创建nfs动态 PV 为例（https://github.com/easzlab/kubeasz/blob/master/docs/setup/08-cluster-storage.md）
3.#####NFS 配置
yum -y install nfs-utils rpcbind
echo '/xvdc 10.2.0.0/16(rw,no_root_squash,no_all_squash,sync)' >> /etc/exports
systemctl restart rpcbind
systemctl restart nfs
systemctl enable rpcbind
systemctl enable nfs
cat /etc/firewalld/zones/public.xml
<service name="rpcbind"/>
<service name="mountd"/>
<service name="nfs"/>
<port protocol="tcp" port="111"/>
<port protocol="tcp" port="2049"/>
<port protocol="udp" port="111"/>
<port protocol="udp" port="2049"/>
/usr/bin/firewall-cmd --reload
#客户端查看可挂载资源 showmount -e 服务器ip10.2.33.33
###nfs-client-provisioner https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
4.安装服务器修改集群配置文件 /etc/ansible/roles/cluster-storage/defaults 以下内容
[root@master-01 defaults]# cat main.yml
# 动态存储类型, 目前支持自建nfs和aliyun_nas
storage:
# nfs server 参数
nfs:
enabled: "yes" #修改为yes
server: "10.2.1.190" #修改为实际nfs的ip
server_path: "/data" #修改为实际nfs共享目录
storage_class: "nfs-es"
provisioner_name: "nfs-provisioner-01"
# aliyun_nas 参数
aliyun_nas:
enabled: "no"
server: "xxxxxxxxxxx.cn-hangzhou.nas.aliyuncs.com"
server_path: "/"
storage_class: "class-aliyun-nas-01"
controller_name: "aliyun-nas-controller-01"
[root@master-01 defaults]# ansible-playbook /etc/ansible/roles/cluster-storage/cluster-storage.yml
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [cluster-storage : 准备nfs-client 配置目录] ***********************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [cluster-storage : 生成nfs-client动态存储配置] **********************************************************************************************************************************************************************************************************************************
ok: [localhost]
TASK [cluster-storage : 开始部署nfs-client动态存储] **********************************************************************************************************************************************************************************************************************************
changed: [localhost]
PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************
localhost : ok=4 changed=1 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0
# 执行成功后验证
[root@master-01 defaults]# kubectl get pod --all-namespaces |grep nfs-prov
kube-system nfs-provisioner-01-6c5f4dff58-4zw9l 1/1 Running 0 4m43s
5.安装 ELK
5.1安装elaticsearch
[root@master-01 elk]# cat elaticsearch.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
generation: 1
labels:
app: elasticsearch-logging
version: v1
name: elasticsearch
namespace: elastic
spec:
minReadySeconds: 10
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: elasticsearch-logging
version: v1
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: elasticsearch-logging
version: v1
spec:
affinity:
nodeAffinity:
containers:
- env:
- name: discovery.type
value: single-node
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
- name: MINIMUM_MASTER_NODES
value: "1"
image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0-amd64
imagePullPolicy: IfNotPresent
name: elasticsearch-logging
ports:
- containerPort: 9200
name: db
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: "1"
memory: 1Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: es-persistent-storage
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: user-1-registrysecret
initContainers:
- command:
- /sbin/sysctl
- -w
- vm.max_map_count=262144
image: alpine:3.6
imagePullPolicy: IfNotPresent
name: elasticsearch-logging-init
resources:
securityContext:
privileged: true
procMount: Default
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /opt/paas/hanju/es_data
type: ""
name: es-persistent-storage
---
apiVersion: v1
kind: Service
metadata:
namespace: elastic
name: elasticsearch
labels:
app: elasticsearch-logging
spec:
type: NodePort
ports:
- port: 9200
targetPort: 9200
name: elasticsearch
selector:
app: elasticsearch-logging
5.2安装logstash
[root@master-01 elk]# cat logstash.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: elastic
data:
logstash.conf: |-
input {
beats {
port => 5044
client_inactivity_timeout => 36000
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
namespace: elastic
labels:
name: logstash
spec:
replicas: 1
selector:
matchLabels:
name: logstash
template:
metadata:
labels:
app: logstash
name: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.12.0
ports:
- containerPort: 5044
protocol: TCP
# - containerPort: 9600
# protocol: TCP
volumeMounts:
- name: logstash-config
#mountPath: /usr/share/logstash/logstash-simple.conf
#mountPath: /usr/share/logstash/config/logstash-sample.conf
mountPath: /usr/share/logstash/pipeline/logstash.conf
subPath: logstash.conf
#ports:
# - containerPort: 80
# protocol: TCP
volumes:
- name: logstash-config
configMap:
#defaultMode: 0644
name : logstash-config
---
apiVersion: v1
kind: Service
metadata:
namespace: elastic
name: logstash
labels:
app: logstash
spec:
type: ClusterIP
ports:
- port: 5044
name: logstash
selector:
app: logstash
5.3安装kibana
[root@master-01 elk]# cat kibana.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: elastic
labels:
name: kibana
spec:
replicas: 1
selector:
matchLabels:
name: kibana
template:
metadata:
labels:
app: kibana
name: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.12.0
ports:
- containerPort: 5601
protocol: TCP
env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch:9200
---
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: elastic
spec:
ports:
- protocol: TCP
port: 5601
targetPort: 5601
selector:
app: kibana
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana
namespace: elastic
spec:
rules:
- host: 10.2.1.194
http:
paths:
- path: /
backend:
serviceName: kibana
servicePort: 5601
5.4安装 nginx filebeat
[root@master-01 elk]# cat nginx_With_filebeat_to_logstash.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config-to-logstash
namespace: elastic
data:
filebeat.yml: |-
filebeat.inputs:
- type: log
paths:
- /logm/*.log
output.logstash:
hosts: [logstash:5044]
---
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-conf
namespace: elastic
data:
nginx.conf: |-
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] ' '"$request" $status $bytes_sent ' '"$http_referer" "$http_user_agent" ' '"$gzip_ratio"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: elastic
labels:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
name: nginx
template:
metadata:
labels:
app: nginx
name: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- name: logm
mountPath: /var/log/nginx/
- name: nginx-conf
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.12.0
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
volumeMounts:
- mountPath: /logm
name: logm
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
volumes:
- name: logm
emptyDir:
- name: config
configMap:
defaultMode: 0640
name: filebeat-config-to-logstash
- name: nginx-conf
configMap:
defaultMode: 0640
name: nginx-conf
按顺序安装即可，ClusterIP模式，配置ingress使其外部可访问
错误1：[root@master-01 es-cluster]# helm install es-cluster --namespace elastic -f es-values.yaml elasticsearch
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec
问题原因： 此命令为helm2安装命令，本机安装的版本为helm3
解决：安装helm2
wget https://get.helm.sh/helm-v2.16.9-linux-amd64.tar.gz
tar -zxvf helm-v2.16.9-linux-amd64.tar.gz
mv linux-amd64/helm /usr/sbin/helm2
错误2：helm Error: could not find tiller
helm2 version
Client: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}
Error: could not find tiller
解决：安装tiller
helm2 init -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.16.9 --stable-repo-url http://mirror.azure.cn/kubernetes/charts/ --service-account tiller --override spec.selector.matchLabels.'name'='tiller',spec.selector.matchLabels.'app'='helm' --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | kubectl apply -f -
错误3：Error: Couldn't load repositories file (/root/.helm/repository/repositories.yaml).
[root@master-01 ~]# helm2 update
Command "update" is deprecated, Use 'helm repo update'
Error: Couldn't load repositories file (/root/.helm/repository/repositories.yaml).
You might need to run `helm init` (or `helm init --client-only` if tiller is already installed)解决： 创建文件
[root@master-01 ~]# echo "" > /root/.helm/repository/repositories.yaml
[root@master-01 ~]# helm2 update
Command "update" is deprecated, Use 'helm repo update'
Error: repository file is out of date
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献10条内容
所有评论(0)