helm安装mysql
导航：一、前提1.版本信息-2.查看方法-3.查看k8s集群节点状态-4.配置docker仓库下载加速二、安装部署helm1.下载软件包-2.安装helm-3.初始化三、使用helm安装mysql1.搜索mysql的helm包-2.准备后端存储-3.准备docker image-4.配置集群内的kube-dns-5.安装mysql的helm包-6.查看mysql的有状态应用、pod、s...
·
导航：
一、前提
1.版本信息-2.查看方法-3.查看k8s集群节点状态-4.配置docker仓库下载加速
二、安装部署helm
1.下载软件包-2.安装helm-3.初始化
三、使用helm安装mysql
1.搜索mysql的helm包-2.准备后端存储-3.准备docker image-4.配置集群内的kube-dns-5.安装mysql的helm包-6.查看mysql的有状态应用、pod、svc、pvc等k8s资源-7.客户端验证
四、故障排除
五、参考资料一、前提
使用helm安装有状态应用mysql主备集群，首先确保你已经搭建好一个3节点的k8s集群(如何搭建k8s集群、k8s集群网络配置flannel请参考百度)。
1.版本信息
k8s: v1.5.2
docker: 1.13.1
flannel:0.7.1
helm: v3.0.02.查看方法
kubectl version
docker version
flanneld --version
helm version3.查看k8s集群节点状态
[root@master ~]# kubectl get node
NAME STATUS AGE
master Ready 4d
node2 Ready 18h
node3 Ready 4m
[root@master ~]# 4.配置docker仓库下载加速
使用163的国内docker仓库，提高docker image的下载速度。
[root@master ~]# cat /etc/docker/daemon.json
{"registry-mirrors": ["http://hub-mirror.c.163.com"]}
[root@master ~]# systemctl restart docker #重启docker服务生效
[root@master ~]# 二、安装部署helm
使用helm的v3版本，v3版本不用配置服务端组件 Tiller。
1.下载软件包
下载helm的二进制包。
下载地址：
https://space.dingtalk.com/s/gwHOAF91cwLOABfU0gPaACBkNjc5YmMwMDUzZmQ0YWQwYjg2MmNhZWRiMGU3YmU1ZA 密码: 5kEK
2.安装helm
使用xshell或者winscp将安装包传输到linux中。
[root@master ~]# tar -tvf helm-v3.0.0-alpha.1-linux-amd64.tar.gz #查看压缩包中的内容
drwxr-xr-x circleci/circleci 0 2019-05-16 05:44 linux-amd64/
-rwxr-xr-x circleci/circleci 39771936 2019-05-16 05:42 linux-amd64/helm
-rw-r--r-- circleci/circleci 11373 2019-05-16 05:44 linux-amd64/LICENSE
-rw-r--r-- circleci/circleci 3159 2019-05-16 05:44 linux-amd64/README.md
[root@master ~]# 将二进制文件复制到$PATH路径下即可使用。
[root@master ~]# which helm #查看helm二进制文件所在的路径，将helm复制到$PATH中即可直接使用，例如将helm放在/usr/bin/helm的路径
/usr/bin/helm
[root@master ~]# helm --help
The Kubernetes package manager
To begin working with Helm, run the 'helm init' command:
$ helm init
This will set up any necessary local configuration.
Common actions from this point include:
- helm search: search for charts
- helm fetch: download a chart to your local directory to view
- helm install: upload the chart to Kubernetes
- helm list: list releases of charts
Environment:
$HELM_HOME set an alternative location for Helm files. By default, these are stored in ~/.helm
$HELM_DRIVER set the backend storage driver. Values are: configmap, secret, memory
$HELM_NO_PLUGINS disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.
$KUBECONFIG set an alternative Kubernetes configuration file (default "~/.kube/config")
Usage:
helm [command]
Available Commands:
chart push, pull, tag, or remove Helm charts
completion Generate autocompletions script for the specified shell (bash or zsh)
create create a new chart with the given name
dependency manage a chart's dependencies
get download a named release
help Help about any command
history fetch release history
home displays the location of HELM_HOME
init initialize Helm client
install install a chart
lint examines a chart for possible issues
list list releases
package package a chart directory into a chart archive
plugin add, list, or remove Helm plugins
pull download a chart from a repository and (optionally) unpack it in local directory
registry login to or logout from a registry
repo add, list, remove, update, and index chart repositories
rollback roll back a release to a previous revision
search search for a keyword in charts
show inspect a chart
status displays the status of the named release
template locally render templates
test test a release or cleanup test artifacts
uninstall uninstall a release
upgrade upgrade a release
verify verify that a chart at the given path has been signed and is valid
version print the client version information
Flags:
--debug enable verbose output
-h, --help help for helm
--home string location of your Helm config. Overrides $HELM_HOME (default "/root/.helm")
--kube-context string name of the kubeconfig context to use
--kubeconfig string path to the kubeconfig file
-n, --namespace string namespace scope for this request
Use "helm [command] --help" for more information about a command.
[root@master ~]# 3.初始化
使用helm前，需要初始化。
[root@docker repository]# helm init --stable-repo-url=https://apphub.aliyuncs.com #配置helm使用的仓库为国内阿里云地址，国外的访问不了，而且比较慢
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://apphub.aliyuncs.com
$HELM_HOME has been configured at /root/.helm.
Happy Helming!
[root@docker repository]# ls
cache local repositories.yaml
[root@docker repository]# cat repositories.yaml #helm初始化完后会有一个配置文件生成，下面是repositories.yaml的内容
apiVersion: v1
generated: "2019-08-25T07:08:38.917953914+08:00"
repositories:
- caFile: ""
cache: /root/.helm/repository/cache/stable-index.yaml
certFile: ""
keyFile: ""
name: stable
password: ""
url: https://apphub.aliyuncs.com
username: ""三、使用helm安装mysql
使用helm安装有状态应用mysql，使用到的docker镜像有：
docker.io/mysql:latest
docker.io/ist0ne/xtrabackup:latest构建mysql的1主2从，主备集群，使用xtrabackup工具进行数据同步。
1.搜索mysql的helm包
[root@docker ~]# helm search mysql #搜索mysql的helm包
NAME CHART VERSION APP VERSION DESCRIPTION
apphub/mysql 6.3.1 8.0.17 Chart to create a Highly available MySQL cluster
apphub/mysqldump 2.6.0 2.4.1 A Helm chart to help backup MySQL databases usi...
apphub/mysqlha 1.0.0 5.7.13 MySQL cluster with a single master and zero or ...
apphub/prometheus-mysql-exporter 0.5.1 v0.11.0 A Helm chart for prometheus mysql exporter with...
stable/mysql 6.3.1 8.0.17 Chart to create a Highly available MySQL cluster
stable/mysqldump 2.6.0 2.4.1 A Helm chart to help backup MySQL databases usi...
stable/mysqlha 1.0.0 5.7.13 MySQL cluster with a single master and zero or ...
stable/prometheus-mysql-exporter 0.5.1 v0.11.0 A Helm chart for prometheus mysql exporter with...
apphub/percona 1.1.0 5.7.17 free, fully compatible, enhanced, open source d...
apphub/percona-xtradb-cluster 1.0.1 5.7.19 free, fully compatible, enhanced, open source d...
apphub/phpmyadmin 3.0.1 4.9.0-1 phpMyAdmin is an mysql administration frontend
stable/percona 1.1.0 5.7.17 free, fully compatible, enhanced, open source d...
stable/percona-xtradb-cluster 1.0.1 5.7.19 free, fully compatible, enhanced, open source d...
stable/phpmyadmin 3.0.1 4.9.0-1 phpMyAdmin is an mysql administration frontend
apphub/mariadb 6.8.1 10.3.17 Fast, reliable, scalable, and easy to use open-...
apphub/mariadb-galera 0.2.1 10.3.17 MariaDB Galera is a multi-master database clust...
stable/mariadb 6.8.1 10.3.17 Fast, reliable, scalable, and easy to use open-...
stable/mariadb-galera 0.2.1 10.3.17 MariaDB Galera is a multi-master database clust...2.准备后端存储
由于mysql的helm包需要使用pv，而开源的k8s集群默认没有配置pv后端存储，因此需要手动添加nfs服务器(如何搭建nfs服务器请参考百度)为后端存储。
nfs配置：
[root@master ~]# cat /etc/exports
/nfsdata/pv001 *(rw,no_root_squash) #添加no_root_squash不压缩root权限，因为容器里会使用root权限写入数据，否则后面会有权限不足的报错
/nfsdata/pv002 *(rw,no_root_squash)
/nfsdata/pv003 *(rw,no_root_squash)
[root@master ~]# ls -ld /nfsdata/ #配置目录的权限为nfsnobody
drwxr-xr-x 5 nfsnobody nfsnobody 45 Aug 26 01:18 /nfsdata/
[root@master ~]# ls -ll /nfsdata/
total 0
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv001
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv002
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv003
[root@master ~]# 创建pv的yaml文件：
[root@master hrr]# cat pv.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv003
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce #accessModes需要和pvc的模式一样
nfs:
path: /nfsdata/pv003
server: 192.168.222.129
[root@master hrr]# 创建pv:
[root@docker hrr]# kubectl create -f pv.yml
persistentvolume "pv003" created
[root@docker hrr]#查看pv：
[root@master hrr]# kubectl get pv #Available表示没有被PVC使用
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pv001 10Gi RWO Retain Available 26s
pv002 10Gi RWO Retain Available 14s
pv003 10Gi RWO Retain Available 3s
[root@master hrr]#
[root@master hrr]# kubectl get pv #Bound表示被PVC使用
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pv001 10Gi RWO Retain Bound default/data-mysql-mysqlha-0 32s
pv002 10Gi RWO Retain Bound default/data-mysql-mysqlha-1 17s
pv003 10Gi RWO Retain Bound default/data-mysql-mysqlha-2 4s
[root@master hrr]#3.准备docker image
在安装mysql的helm包时需要mysql与xtrabackup的docker镜像。
[root@docker docker]# docker pull docker.io/ist0ne/xtrabackup #下载xtrabackup镜像
Using default tag: latest
Trying to pull repository docker.io/ist0ne/xtrabackup ...
latest: Pulling from docker.io/ist0ne/xtrabackup
386a066cd84a: Pull complete
40f175e652e1: Pull complete
5eb6b5905d55: Pull complete
Digest: sha256:e5e5519b874363f1bf35541cef2caed5803645937e29834817be51b30aabc479
Status: Downloaded newer image for docker.io/ist0ne/xtrabackup:latest
[root@docker docker]#
[root@docker docker]# docker pull docker.io/mysql #下载mysql镜像查看本地的镜像：
[root@master hrr]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.222.129:5000/nginx latest 5a3221f0137b 9 days ago 126 MB
docker.io/nginx latest 5a3221f0137b 9 days ago 126 MB
docker.io/mysql latest 62a9f311b99c 11 days ago 445 MB
mysql 5.7.13 62a9f311b99c 11 days ago 445 MB
docker.io/ist0ne/xtrabackup latest c415dbd7af07 2 years ago 265 MB
registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup 1.0 c415dbd7af07 2 years ago 265 MB
192.168.222.129:5000/pod-infrastructure latest f9d5de079539 5 years ago 240 kB
docker.io/kubernetes/pause latest f9d5de079539 5 years ago 240 kB
[root@master hrr]#将镜像打tag，用于后面创建mysql的helm包：
[root@docker hrr]# docker tag docker.io/mysql:latest mysql:5.7.13
[root@docker docker]# docker tag docker.io/ist0ne/xtrabackup:latest registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0将docker镜像导出，用于导入到其它k8s节点中：
[root@master ~]# docker save mysql:5.7.13 -o mysql5713.tgz #将docker镜像导出，名为mysql5713.tgz
[root@master ~]# docker save registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 -o xtrabackup10.tgz
[root@master ~]# scp mysql5713.tgz xtrabackup10.tgz root@192.168.222.130:
root@192.168.222.130's password:
mysql5713.tgz 100% 429MB 35.7MB/s 00:12
xtrabackup10.tgz 100% 260MB 43.4MB/s 00:06
[root@master ~]#在其它k8s节点中导入docker镜像：
[root@node2 ~]# docker load -i mysql5713.tgz #将导出的docker镜像mysql5713.tgz导入
8fa655db5360: Loading layer [==================================================>] 58.48 MB/58.48 MB
f5741d086b76: Loading layer [==================================================>] 338.4 kB/338.4 kB
9f77b78f01a7: Loading layer [==================================================>] 10.44 MB/10.44 MB
7f33ce1066af: Loading layer [==================================================>] 4.472 MB/4.472 MB
1dd5f3e365cf: Loading layer [==================================================>] 1.536 kB/1.536 kB
65430c57aee2: Loading layer [==================================================>] 46.15 MB/46.15 MB
bb1ef34119b2: Loading layer [==================================================>] 32.77 kB/32.77 kB
995fa4bb8afe: Loading layer [==================================================>] 3.584 kB/3.584 kB
bdebd19e878f: Loading layer [==================================================>] 329.8 MB/329.8 MB
79a46f1eb9d0: Loading layer [==================================================>] 5.632 kB/5.632 kB
5b96d5043191: Loading layer [==================================================>] 9.728 kB/9.728 kB
647229410363: Loading layer [==================================================>] 1.536 kB/1.536 kB
Loaded image: mysql:5.7.13
[root@node2 ~]# docker load -i xtrabackup10.tgz
fe4c16cbf7a4: Loading layer [==================================================>] 128.9 MB/128.9 MB
82d548d175dd: Loading layer [==================================================>] 9.216 kB/9.216 kB
f85c58969eb0: Loading layer [==================================================>] 144 MB/144 MB
Loaded image: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0
[root@node2 ~]#node3做同样的操作。
4.配置集群内的kube-dns
5.安装mysql的helm包
[root@master ~]# helm install mysql apphub/mysqlha
NAME: mysql
LAST DEPLOYED: 2019-08-25 09:43:33.16471802 +0800 CST m=+0.898633546
NAMESPACE: default
STATUS: deployed
NOTES:
The MySQL cluster is comprised of 3 MySQL pods: 1 master and 2 slaves. Each instance is accessible within the cluster through:
<pod-name>.mysql-mysqlha
`mysql-mysqlha-0.mysql-mysqlha` is designated as the master and where all writes should be executed against. Read queries can be executed against the `mysql-mysqlha-readonly` service which distributes connections across all MySQL pods.
To connect to your database:
1. Obtain the root password:
kubectl get secret --namespace default mysql-mysqlha -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo
2. Run a pod to use as a client:
kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh
3. To connect to Master service (read/write):
mysql -h mysql-mysqlha-0.mysql-mysqlha -u root -p
4. To connect to slave service (read-only):
mysql -h mysql-mysqlha-readonly -u root -p
[root@master ~]#安装完后，在本地会有mysql的helm包：
[root@master archive]# pwd
/root/.helm/cache/archive
[root@master archive]# ls
guestbook-0.2.0.tgz mysqlha-1.0.0.tgz
[root@master archive]#
[root@master archive]# tar -tvf mysqlha-1.0.0.tgz
-rwxr-xr-x 0/0 378 2019-08-22 13:20 mysqlha/Chart.yaml
-rwxr-xr-x 0/0 2561 2019-08-22 13:20 mysqlha/values.yaml
-rwxr-xr-x 0/0 1216 2019-08-22 13:20 mysqlha/templates/NOTES.txt
-rwxr-xr-x 0/0 961 2019-08-22 13:20 mysqlha/templates/_helpers.tpl
-rwxr-xr-x 0/0 756 2019-08-22 13:20 mysqlha/templates/configmap.yaml
-rwxr-xr-x 0/0 967 2019-08-22 13:20 mysqlha/templates/secret.yaml
-rwxr-xr-x 0/0 10905 2019-08-22 13:20 mysqlha/templates/statefulset.yaml
-rwxr-xr-x 0/0 1203 2019-08-22 13:20 mysqlha/templates/svc.yaml
-rwxr-xr-x 0/0 56 2019-08-22 13:20 mysqlha/OWNERS
-rwxr-xr-x 0/0 6724 2019-08-22 13:20 mysqlha/README.md
[root@master archive]# 将mysqlha-1.0.0.tgz解压，查看文件内容：
[root@master mysqlha]# ls
Chart.yaml OWNERS README.md templates values.yaml
[root@master mysqlha]# cat README.md
# MySQL - Single Master, Multiple Slaves
[MySQL](https://MySQL.org) is one of the most popular database servers in the world. Notable users include Wikipedia, Facebook and Google.
## Introduction
This chart bootstraps a single master and multiple slave MySQL deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager. Largely inspired by this [tutorial](https://kubernetes.io/docs/tutorials/stateful-application/run-replicated-stateful-application/), further work was made to 'production-ize' the example.
## Prerequisites
- Kubernetes 1.6+
- PV provisioner support in the underlying infrastructure
## Installing the Chart
To install the chart with the release name `my-release`:
$ helm install --name my-release <helm-repo>/mysqlha
The command deploys MySQL cluster on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
### Uninstall
To uninstall/delete the `my-release` deployment:
$ helm delete my-release
## Configuration
The following table lists the configurable parameters of the MySQL chart and their default values.
| Parameter | Description | Default |
| ----------------------------------------- | ------------------------------------------------- | -------------------------------------- |
| `mysqlImage` | `mysql` image and tag. | `mysql:5.7.13` |
| `xtraBackupImage` | `xtrabackup` image and tag. | `gcr.io/google-samples/xtrabackup:1.0` |
| `imagePullPolicy` | Image pull policy. | `IfNotPresent` |
| `nameOverride` | `String to partially override mysqlha.fullname template with a string (will prepend the release name)` | `nil` |
| `fullnameOverride` | `String to fully override mysqlha.fullname template with a string` | `nil` |
| `replicaCount` | Number of MySQL replicas | 3 |
| `mysqlRootPassword` | Password for the `root` user. | Randomly generated |
| `mysqlUser` | Username of new user to create. | `nil` |
| `mysqlPassword` | Password for the new user. | Randomly generated |
| `mysqlReplicationUser` | Username for replication user | `repl` |
| `mysqlReplicationPassword` | Password for replication user. | Randomly generated |
| `mysqlDatabase` | Name of the new Database to create | `nil` |
| `configFiles.master.cnf` | Master configuration file | See `values.yaml` |
| `configFiles.slave.cnf` | Slave configuration file | See `values.yaml` |
| `podAnnotations` | Pod annotations | `nil` |
| `schedulerName` | Name of the k8s scheduler (other than default) | `nil` |
| `persistence.enabled` | Create a volume to store data | true |
| `persistence.size` | Size of persistent volume claim | 10Gi |
| `persistence.storageClass` | Type of persistent volume claim | `nil` |
| `persistence.accessModes` | Persistent volume access modes | `[ReadWriteOnce]` |
| `persistence.annotations` | Persistent volume annotations | `{}` |
| `resources` | CPU/Memory resource requests/limits | Memory: `128Mi`, CPU: `100m` |
| `metrics.enabled` | Start a side-car prometheus exporter | false |
| `metrics.image` | Exporter image | `prom/mysqld-exporter` |
| `metrics.imageTag` | Exporter image | `v0.10.0` |
| `metrics.imagePullPolicy` | Exporter image pull policy | `IfNotPresent` |
| `metrics.resources` | Exporter resource requests/limit | See `values.yaml` |
| `metrics.livenessProbe.initialDelaySeconds` | Delay before metrics liveness probe is initiated | 15 |
| `metrics.livenessProbe.timeoutSeconds` | When the probe times out | 5 |
| `metrics.readinessProbe.initialDelaySeconds` | Delay before metrics readiness probe is initiated | 5 |
| `metrics.readinessProbe.timeoutSeconds` | When the probe times out | 1 |
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
## Persistence
The [MySQL](https://hub.docker.com/_/mysql/) image stores the MySQL data and configurations at the `/var/lib/mysql` path of the container.
By default persistence is enabled, and a PersistentVolumeClaim is created and mounted in that directory. As a result, a persistent volume will need to be defined:
# https://kubernetes.io/docs/user-guide/persistent-volumes/#azure-disk
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: kubernetes.io/azure-disk
parameters:
skuName: Premium_LRS
location: westus
In order to disable this functionality you can change the values.yaml to disable persistence and use an emptyDir instead.查看values.yaml文件：
[root@master mysqlha]# cat values.yaml
## mysql image version
## ref: https://hub.docker.com/r/library/mysql/tags/
##
mysqlImage: mysql:5.7.13 #前面下载的mysql镜像
xtraBackupImage: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 #前面下载的xtrabackup镜像
## Specify an imagePullPolicy (Required)
## It's recommended to change this to 'Always' if the image tag is 'latest'
## ref: http://kubernetes.io/docs/user-guide/images/#updating-images
##
imagePullPolicy: IfNotPresent
## String to partially override orangehrm.fullname template (will maintain the release name)
##
# nameOverride: ""
## String to fully override orangehrm.fullname template
##
# fullnameOverride: ""
mysqlha:
replicaCount: 3
## Password for MySQL root user
##
# mysqlRootPassword: ## Default: random 10 character string
## Username/password for MySQL replication user
##
mysqlReplicationUser: repl
# mysqlReplicationPassword:
## Create a database user
##
# mysqlUser:
# mysqlPassword: ## Default: random 10 character string
## Allow unauthenticated access, uncomment to enable
##
# mysqlAllowEmptyPassword: true
## Create database with name and grant all permissions to user on startup, if needed
# mysqlDatabase:
## Configuration files for the master and slaves
##
configFiles:
master.cnf: |
# Apply this config only on the master.
[mysqld]
log-bin
skip_name_resolve
slave.cnf: |
# Apply this config only on slaves.
[mysqld]
super-read-only
skip_name_resolve
podAnnotations: {}
## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:
## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
enabled: true
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, azure-disk on
## Azure, standard on GKE, AWS & OpenStack)
##
# storageClass: "-"
accessModes:
- ReadWriteOnce
size: 10Gi
annotations: {}
resources:
requests:
cpu: 100m
memory: 128Mi
metrics:
enabled: false
image: prom/mysqld-exporter
imageTag: v0.10.0
annotations: {}
livenessProbe:
initialDelaySeconds: 15
timeoutSeconds: 5
readinessProbe:
initialDelaySeconds: 5
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 100Mi
[root@master mysqlha]# 6.查看mysql的有状态应用、pod、svc、pvc等k8s资源
[root@master hrr]# kubectl get statefulset
NAME DESIRED CURRENT AGE
mysql-mysqlha 3 1 18s
[root@master hrr]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 4d
mysql-mysqlha None <none> 3306/TCP 27s
mysql-mysqlha-readonly 10.254.54.188 <none> 3306/TCP 27s
[root@master hrr]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-mysqlha-0 2/2 Running 0 45s
mysql-mysqlha-1 2/2 Running 0 24s
mysql-mysqlha-2 1/2 Running 0 4s
[root@master hrr]# kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pv001 10Gi RWO Retain Bound default/data-mysql-mysqlha-0 2m
pv002 10Gi RWO Retain Bound default/data-mysql-mysqlha-2 2m
pv003 10Gi RWO Retain Bound default/data-mysql-mysqlha-1 1m
[root@master hrr]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
data-mysql-mysqlha-0 Bound pv001 10Gi RWO 51s
data-mysql-mysqlha-1 Bound pv003 10Gi RWO 51s
data-mysql-mysqlha-2 Bound pv002 10Gi RWO 51s
[root@master hrr]# kubectl get configmap
NAME DATA AGE
mysql-mysqlha 4 54s
[root@master hrr]# kubectl get secret
NAME TYPE DATA AGE
mysql-mysqlha Opaque 2 51m
mysql.v1 helm.sh/release 1 51m
[root@master hrr]#
[root@master hrr]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-mysqlha-0 2/2 Running 0 1m 172.17.93.2 node3
mysql-mysqlha-1 2/2 Running 0 41s 172.17.78.2 node2
mysql-mysqlha-2 2/2 Running 0 21s 172.17.86.2 master查看第一个节点的docker容器
[root@master hrr]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
41ec983706b8 registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 "/bin/bash -c 'set..." 25 seconds ago Up 24 seconds k8s_xtrabackup.516a4a61_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_72aba107
691066dcf870 mysql:5.7.13 "docker-entrypoint..." 25 seconds ago Up 24 seconds k8s_mysql.b9eef278_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_894896de
d83f031101ef 192.168.222.129:5000/pod-infrastructure:latest "/pause" 26 seconds ago Up 25 seconds k8s_POD.62b22cd9_mysql-mysqlha-2_default_387e4cdb-c772-11e9-83c0-000c296be0a4_c46314b7查看第二个节点的docker容器
[root@node2 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
211a5deed23f registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 "/bin/bash -c 'set..." 52 seconds ago Up 51 seconds k8s_xtrabackup.516a4a61_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_982995a3
51d9e7cd3d1d mysql:5.7.13 "docker-entrypoint..." 52 seconds ago Up 51 seconds k8s_mysql.b9eef278_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_3acd8bf2
71bb21654249 192.168.222.129:5000/pod-infrastructure:latest "/pause" 52 seconds ago Up 51 seconds k8s_POD.62b22cd9_mysql-mysqlha-1_default_2c7419c8-c772-11e9-83c0-000c296be0a4_cce640c3
[root@node2 ~]# 查看第三个节点的docker容器
[root@node3 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d159b422bcdf registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 "/bin/bash -c 'set..." About a minute ago Up About a minute k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_63f2a879
6aae997c995d mysql:5.7.13 "docker-entrypoint..." About a minute ago Up About a minute k8s_mysql.b9eef278_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_7dee904b
40695f86aaf7 192.168.222.129:5000/pod-infrastructure:latest "/pause" About a minute ago Up About a minute k8s_POD.62b22cd9_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_58c67c1e
[root@node3 ~]#查看pvc里的数据
[root@master hrr]# ll /nfsdata/pv001
total 4
drwxr-xr-x 6 polkitd input 4096 Aug 26 03:54 mysql
[root@master hrr]# ll /nfsdata/pv001/mysql/
total 177164
-rw-r----- 1 polkitd input 56 Aug 26 03:54 auto.cnf
-rw-r----- 1 polkitd input 3091158 Aug 26 03:54 binlog.000001
-rw-r----- 1 polkitd input 155 Aug 26 03:54 binlog.000002
-rw-r----- 1 polkitd input 32 Aug 26 03:54 binlog.index
-rw------- 1 polkitd input 1676 Aug 26 03:54 ca-key.pem
-rw-r--r-- 1 polkitd input 1112 Aug 26 03:54 ca.pem
-rw-r--r-- 1 polkitd input 1112 Aug 26 03:54 client-cert.pem
-rw------- 1 polkitd input 1680 Aug 26 03:54 client-key.pem
-rw-r----- 1 polkitd input 5419 Aug 26 03:54 ib_buffer_pool
-rw-r----- 1 polkitd input 12582912 Aug 26 03:54 ibdata1
-rw-r----- 1 polkitd input 50331648 Aug 26 03:54 ib_logfile0
-rw-r----- 1 polkitd input 50331648 Aug 26 03:54 ib_logfile1
-rw-r----- 1 polkitd input 12582912 Aug 26 03:55 ibtmp1
drwxr-x--- 2 polkitd input 187 Aug 26 03:54 #innodb_temp
drwxr-x--- 2 polkitd input 143 Aug 26 03:54 mysql
-rw-r----- 1 polkitd input 29360128 Aug 26 03:54 mysql.ibd
drwxr-x--- 2 polkitd input 8192 Aug 26 03:54 performance_schema
-rw------- 1 polkitd input 1680 Aug 26 03:54 private_key.pem
-rw-r--r-- 1 polkitd input 452 Aug 26 03:54 public_key.pem
-rw-r--r-- 1 polkitd input 1112 Aug 26 03:54 server-cert.pem
-rw------- 1 polkitd input 1680 Aug 26 03:54 server-key.pem
drwxr-x--- 2 polkitd input 28 Aug 26 03:54 sys
-rw-r----- 1 polkitd input 12582912 Aug 26 03:54 undo_001
-rw-r----- 1 polkitd input 10485760 Aug 26 03:54 undo_002查看svc
[root@master hrr]# kubectl describe svc mysql-mysqlha
Name: mysql-mysqlha
Namespace: default
Labels: app=mysql-mysqlha
chart=mysqlha-1.0.0
heritage=Helm
release=mysql
Selector: app=mysql-mysqlha
Type: ClusterIP
IP: None
Port: mysql-mysqlha 3306/TCP
Endpoints: 172.17.78.2:3306,172.17.86.2:3306,172.17.93.2:3306
Session Affinity: None
No events.
[root@master hrr]# kubectl describe svc mysql-mysqlha-readonly
Name: mysql-mysqlha-readonly
Namespace: default
Labels: app=mysql-mysqlha
chart=mysqlha-1.0.0
heritage=Helm
release=mysql
Selector: app=mysql-mysqlha
Type: ClusterIP
IP: 10.254.54.188
Port: mysql-mysqlha 3306/TCP
Endpoints: 172.17.78.2:3306,172.17.86.2:3306,172.17.93.2:3306
Session Affinity: None
No events.
[root@master hrr]# 查看有状态应用stateful
[root@master hrr]# kubectl get statefulset -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
creationTimestamp: 2019-08-25T19:54:18Z
generation: 1
labels:
app: mysql-mysqlha
chart: mysqlha-1.0.0
heritage: Helm
release: mysql
name: mysql-mysqlha
namespace: default
resourceVersion: "94760"
selfLink: /apis/apps/v1beta1/namespaces/default/statefulsets/mysql-mysqlha
uid: 1fa6f230-c772-11e9-83c0-000c296be0a4
spec:
replicas: 3
selector:
matchLabels:
app: mysql-mysqlha
serviceName: mysql-mysqlha
template:
metadata:
creationTimestamp: null
labels:
app: mysql-mysqlha
spec:
containers:
- env:
- name: MYSQL_DATABASE
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: mysql-root-password
name: mysql-mysqlha
- name: MYSQL_REPLICATION_USER
value: repl
- name: MYSQL_REPLICATION_PASSWORD
valueFrom:
secretKeyRef:
key: mysql-replication-password
name: mysql-mysqlha
image: mysql:5.7.13
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- /bin/sh
- -c
- mysqladmin ping -h 127.0.0.1 -u root -p${MYSQL_ROOT_PASSWORD}
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: mysql
ports:
- containerPort: 3306
name: mysql
protocol: TCP
readinessProbe:
exec:
command:
- /bin/sh
- -c
- MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
- mysql -h 127.0.0.1 -u root -e "SELECT 1"
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 128Mi
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/lib/mysql
name: data
subPath: mysql
- mountPath: /etc/mysql/conf.d
name: conf
- args:
- -c
- |
set -ex
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 5; done
# Create replication user
cd /mnt/scripts
# file exists and is not empty with -s
if [[ -s create-replication-user.sh ]]; then
ls -la
./create-replication-user.sh
fi
cd /var/lib/mysql
# Determine binlog position of cloned data, if any.
if [[ -f xtrabackup_slave_info ]]; then
# XtraBackup already generated a partial "CHANGE MASTER TO" query
# because we're cloning from an existing slave.
cp xtrabackup_slave_info change_master_to.sql.in
elif [[ -f xtrabackup_binlog_info ]]; then
# We're cloning directly from master. Parse binlog position.
[[ $(cat xtrabackup_binlog_info) =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
# Check if we need to complete a clone by starting replication.
if [[ -f change_master_to.sql.in ]]; then
# In case of container restart, attempt this at-most-once.
cp change_master_to.sql.in change_master_to.sql.orig
mysql -h 127.0.0.1 --verbose<<EOF
STOP SLAVE IO_THREAD;
$(<change_master_to.sql.orig),
MASTER_HOST='mysql-mysqlha-0.mysql-mysqlha',
MASTER_USER='${MYSQL_REPLICATION_USER}',
MASTER_PASSWORD='${MYSQL_REPLICATION_PASSWORD}',
MASTER_CONNECT_RETRY=10;
START SLAVE;
EOF
fi
# Start a server to send backups when requested by peers.
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=${MYSQL_REPLICATION_USER} --password=${MYSQL_REPLICATION_PASSWORD}"
command:
- /bin/bash
env:
- name: MYSQL_PWD
valueFrom:
secretKeyRef:
key: mysql-root-password
name: mysql-mysqlha
- name: MYSQL_REPLICATION_USER
value: repl
- name: MYSQL_REPLICATION_PASSWORD
valueFrom:
secretKeyRef:
key: mysql-replication-password
name: mysql-mysqlha
image: registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0
imagePullPolicy: IfNotPresent
name: xtrabackup
ports:
- containerPort: 3307
name: xtrabackup
protocol: TCP
resources:
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /var/lib/mysql
name: data
subPath: mysql
- mountPath: /etc/mysql/conf.d
name: conf
- mountPath: /mnt/scripts
name: scripts
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: conf
- configMap:
defaultMode: 420
name: mysql-mysqlha
name: config-map
- emptyDir: {}
name: scripts
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
status:
phase: Pending
status:
replicas: 3
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""
[root@master hrr]# 7.客户端验证
创建mysql-client容器，参考前面“ helm install mysql apphub/mysqlha”中的
[root@master hrr]# kubectl run mysql-client --image=mysql:5.7.13 -it --rm --restart='Never' --namespace default -- /bin/sh
Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs: error dialing backend: dial tcp: lookup node3 on 192.168.222.2:53: no such host
Error from server: Get https://node3:10250/containerLogs/default/mysql-client/mysql-client: dial tcp: lookup node3 on 192.168.222.2:53: no such host
[root@master hrr]#
[root@master hrr]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-client 1/1 Running 0 48s 172.17.93.3 node3
mysql-mysqlha-0 2/2 Running 0 54m 172.17.93.2 node3
mysql-mysqlha-1 2/2 Running 0 53m 172.17.78.2 node2
mysql-mysqlha-2 2/2 Running 0 53m 172.17.86.2 master
[root@master hrr]# 在node3上查看
[root@node3 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8261abe363b0 mysql:5.7.13 "docker-entrypoint..." 56 seconds ago Up 55 seconds k8s_mysql-client.d32e2e5_mysql-client_default_90905092-c779-11e9-83c0-000c296be0a4_0d4f202f
1fac37badaa8 192.168.222.129:5000/pod-infrastructure:latest "/pause" 56 seconds ago Up 56 seconds k8s_POD.cace165_mysql-client_default_90905092-c779-11e9-83c0-000c296be0a4_a0f9ab22
d159b422bcdf registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 "/bin/bash -c 'set..." 54 minutes ago Up 54 minutes k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_63f2a879
6aae997c995d mysql:5.7.13 "docker-entrypoint..." 54 minutes ago Up 54 minutes k8s_mysql.b9eef278_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_7dee904b
40695f86aaf7 192.168.222.129:5000/pod-infrastructure:latest "/pause" 54 minutes ago Up 54 minutes k8s_POD.62b22cd9_mysql-mysqlha-0_default_1fbcfe77-c772-11e9-83c0-000c296be0a4_58c67c1e
[root@node3 ~]# 写入数据
[root@node3 ~]# docker exec -ti 8261abe363b0 sh
# mysql -h172.17.93.2 -uroot
ERROR 1045 (28000): Access denied for user 'root'@'172.17.93.3' (using password: NO)
# mysql -h172.17.93.2 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1018
Server version: 8.0.17 MySQL Community Server - GPL
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.01 sec)
mysql> CREATE DATABASE test;
Query OK, 1 row affected (0.02 sec)
mysql> CREATE TABLE test.messages (message VARCHAR(250));
Query OK, 0 rows affected (0.03 sec)
mysql> INSERT INTO test.messages VALUES ('hello');
Query OK, 1 row affected (0.03 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from messages;
+---------+
| message |
+---------+
| hello |
+---------+
1 row in set (0.01 sec)
mysql> exit
Bye
# 连接svc查看，这个是read-only的，可以看到是轮询的，查询了2次才到有test中的库去
# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1058
Server version: 8.0.17 MySQL Community Server - GPL
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
mysql> exit
Bye
# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1054
Server version: 8.0.17 MySQL Community Server - GPL
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.01 sec)
mysql> exit
Bye
# mysql -h10.254.54.188 -uroot -pghScMZT1bTJN
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1072
Server version: 8.0.17 MySQL Community Server - GPL
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> exit
Bye
#四、故障排除
1.helm init报错
故障现象：
[root@docker ~]# helm init
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Error: https://kubernetes-charts.storage.googleapis.com is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: net/http: TLS handshake timeout
[root@docker ~]# 可能原因：
可能国内访问不了https://kubernetes-charts.storage.googleapis.com/index.yaml，并且网速慢。
解决方法：
在初始化时，使用--stable-repo-url参数，将仓库地址改为阿里云的helm仓库地址。helm init --stable-repo-url=https://apphub.aliyuncs.com
2.Pending、PersistentVolumeClaim is not bound
故障现象：
[root@docker ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-mysqlha-0 0/2 Pending 0 35s
nginx 1/1 Running 0 2m
[root@docker ~]#
[root@docker ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 4s 8 {default-scheduler } Warning FailedScheduling SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "data-mysql-mysqlha-0", which is unexpected.可能原因：
第一次使用helm安装mysql时，没有提前创建pv，导致报pvc没有bound的错。
解决方法：
查看pvc的模式，创建pv，创建过程请参考前面。
3.RunContainerError、Failed to pull image "mysql:5.7.13"
故障现象：
[root@docker hrr]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-mysqlha-0 0/2 RunContainerError 0 19s
nginx 1/1 Running 1 1h
[root@docker hrr]#
[root@docker ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
18m 5m 49 {default-scheduler } Warning FailedScheduling SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "data-mysql-mysqlha-0", which is unexpected.
4m 4m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-mysqlha-0 to master
4m 4m 1 {kubelet master} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
4m 4m 1 {kubelet master} spec.containers{mysql} Normal Pulling pulling image "mysql:5.7.13"
10s 10s 1 {kubelet master} spec.containers{mysql} Warning Failed Failed to pull image "mysql:5.7.13": Get https://registry-1.docker.io/v2/library/mysql/manifests/5.7.13: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fmysql%3Apull&service=registry.docker.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
10s 10s 1 {kubelet master} spec.containers{xtrabackup} Normal Pulling pulling image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0"
9s 9s 1 {kubelet master} spec.containers{xtrabackup} Warning Failed Failed to pull image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0": image pull failed for registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0, this may be because there are no credentials on this request. details: (repository registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup not found: does not exist or no pull access)
9s 9s 1 {kubelet master} Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with ErrImagePull: "Get https://registry-1.docker.io/v2/library/mysql/manifests/5.7.13: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fmysql%3Apull&service=registry.docker.io: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
, failed to "StartContainer" for "xtrabackup" with ErrImagePull: "image pull failed for registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0, this may be because there are no credentials on this request. details: (repository registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup not found: does not exist or no pull access)"
]
8s 8s 1 {kubelet master} spec.containers{mysql} Normal BackOff Back-off pulling image "mysql:5.7.13"
8s 8s 1 {kubelet master} spec.containers{xtrabackup} Normal BackOff Back-off pulling image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0"
8s 8s 1 {kubelet master} Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with ImagePullBackOff: "Back-off pulling image \"mysql:5.7.13\""
, failed to "StartContainer" for "xtrabackup" with ImagePullBackOff: "Back-off pulling image \"registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0\""
]
[root@docker hrr]# 可能原因：
下载不到docker image，本地没有mysql镜像。
解决方法：
手动下载mysql镜像，将镜像docker tag为配置中的名字，如何手动下载，请参考上面。
4.CrashLoopBackOff、permission denied
故障现象：
[root@docker docker]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-mysqlha-0 0/2 RunContainerError 0 9s
nginx 1/1 Running 2 1h
[root@docker docker]#
[root@docker docker]# kubectl describe pod mysql-mysqlha-0
……
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15s 15s 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-mysqlha-0 to master
14s 14s 1 {kubelet master} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
14s 3s 3 {kubelet master} spec.containers{mysql} Normal Pulled Container image "mysql:5.7.13" already present on machine
14s 3s 3 {kubelet master} spec.containers{xtrabackup} Normal Pulled Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
14s 3s 3 {kubelet master} Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
[root@docker docker]#
[root@docker docker]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-mysqlha-0 1/2 CrashLoopBackOff 2 5m
nginx 1/1 Running 2 1h
[root@docker docker]#
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5m 5m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-mysqlha-0 to master
1m 1m 1 {kubelet master} Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
5m 44s 23 {kubelet master} Warning FailedSync Error syncing pod, skipping: [failed to "StartContainer" for "mysql" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
, failed to "StartContainer" for "xtrabackup" with RunContainerError: "GenerateRunContainerOptions: mkdir /var/lib/kubelet/pods/18f319d5-c6d2-11e9-baf7-000c296be0a4/volumes/kubernetes.io~nfs/pv001/mysql: permission denied"
]
32s 32s 1 {kubelet master} spec.containers{mysql} Normal Started Started container with docker id d2f5e2c645b8
32s 32s 1 {kubelet master} spec.containers{xtrabackup} Normal Started Started container with docker id 6f2fb901a587
5m 32s 25 {kubelet master} spec.containers{xtrabackup} Normal Pulled Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
32s 32s 1 {kubelet master} spec.containers{mysql} Normal Created Created container with docker id d2f5e2c645b8; Security:[seccomp=unconfined]
32s 32s 1 {kubelet master} spec.containers{xtrabackup} Normal Created Created container with docker id 6f2fb901a587; Security:[seccomp=unconfined]
30s 30s 1 {kubelet master} spec.containers{mysql} Normal Created Created container with docker id 104f64db6e8c; Security:[seccomp=unconfined]
30s 30s 1 {kubelet master} spec.containers{mysql} Normal Started Started container with docker id 104f64db6e8c
28s 26s 3 {kubelet master} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 10s restarting failed container=mysql pod=mysql-mysqlha-0_default(18f319d5-c6d2-11e9-baf7-000c296be0a4)"
5m 11s 27 {kubelet master} spec.containers{mysql} Normal Pulled Container image "mysql:5.7.13" already present on machine
5m 11s 5 {kubelet master} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
11s 11s 1 {kubelet master} spec.containers{mysql} Normal Created Created container with docker id ec8f0827294b; Security:[seccomp=unconfined]
11s 11s 1 {kubelet master} spec.containers{mysql} Normal Started Started container with docker id ec8f0827294b
28s 7s 5 {kubelet master} spec.containers{mysql} Warning BackOff Back-off restarting failed docker container
10s 7s 2 {kubelet master} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 20s restarting failed container=mysql pod=mysql-mysqlha-0_default(18f319d5-c6d2-11e9-baf7-000c296be0a4)"
[root@docker docker]#
[root@docker docker]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
24a5e40cfbe6 mysql:5.7.13 "docker-entrypoint..." About a minute ago Exited (1) About a minute ago k8s_mysql.b9eef278_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_f37f9154
6f2fb901a587 registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 "/bin/bash -c 'set..." 5 minutes ago Up 5 minutes k8s_xtrabackup.516a4a61_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_4c633d64
61a82b9ad65a 192.168.222.129:5000/pod-infrastructure:latest "/pause" 10 minutes ago Up 10 minutes k8s_POD.62b22cd9_mysql-mysqlha-0_default_18f319d5-c6d2-11e9-baf7-000c296be0a4_9f7337a5
dc6071e8f613 192.168.222.129:5000/nginx "nginx -g 'daemon ..." 26 minutes ago Up 26 minutes k8s_nginx.3556dff8_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_9d257b60
15929311253f 192.168.222.129:5000/pod-infrastructure:latest "/pause" 26 minutes ago Up 26 minutes k8s_POD.cace165_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_d9ad1e0f
48f464ad9c9b 192.168.222.129:5000/pod-infrastructure:latest "/pause" 37 minutes ago Exited (2) 26 minutes ago k8s_POD.cace165_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_eb3bebdb
c01c41763b06 192.168.222.129:5000/nginx "nginx -g 'daemon ..." About an hour ago Exited (0) 37 minutes ago k8s_nginx.3556dff8_nginx_default_30a915ea-c6c5-11e9-bbd7-000c296be0a4_0848bc8b
[root@docker docker]# docker logs 24a5e40cfbe6
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
[root@docker docker]#可能原因：
docker对pvc没有权限，由于这里使用的是nfs作后端存储，所以可能是nfs目录的权限没放开。
解决方法：
将nfs目录权限设置为chown nfsnobody:nfsnobody -R /nfsdata，并且/etc/exportfs中设置no_root_squash，不压缩root权限，即在docker可以使用任何用户写入数据，具体操作步骤请参考上面。
5.MissingClusterDNS
故障现象：
[root@master ~]# kubectl describe pod mysql-mysqlha-0
……
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
56m 56m 1 {default-scheduler } Normal Scheduled Successfully assigned mysql-mysqlha-0 to master
56m 56m 3 {kubelet master} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.
56m 56m 1 {kubelet master} spec.containers{mysql} Normal Pulled Container image "mysql:5.7.13" already present on machine
56m 56m 1 {kubelet master} spec.containers{mysql} Normal Created Created container with docker id 31013a7822ae; Security:[seccomp=unconfined]
56m 56m 1 {kubelet master} spec.containers{mysql} Normal Started Started container with docker id 31013a7822ae
56m 56m 1 {kubelet master} spec.containers{xtrabackup} Normal Pulled Container image "registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0" already present on machine
56m 56m 1 {kubelet master} spec.containers{xtrabackup} Normal Created Created container with docker id a5f2f49c1eef; Security:[seccomp=unconfined]
56m 56m 1 {kubelet master} spec.containers{xtrabackup} Normal Started Started container with docker id a5f2f49c1eef可能原因：
没有配置kube-dns。
解决方法：
请参考上面配置kube-dns。
故障现象：
可能原因：
解决方法：
五、参考资料
K8s Helm Charts 一键安装云原生应用指南：
https://mp.weixin.qq.com/s/qeLA2F4vNGUDYVdQSHIWtQ
mysql有状态服务部署：
https://segmentfault.com/a/1190000012244714
记k8s的一次pv&pvc创建过程：
https://staight.github.io/2018/08/18/%E8%AE%B0k8s%E7%9A%84%E4%B8%80%E6%AC%A1pv&pvc%E5%88%9B%E5%BB%BA%E8%BF%87%E7%A8%8B/
nfs挂载Permission denied问题解决：
https://blog.csdn.net/weixin_42256397/article/details/97283989
K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)