K8S集群java项目上云部署
一、项目流程拓扑：
1、项目主机规划
|
编号 |
主机名 |
主机ip |
服务 |
域名 |
|
1 |
Master01 |
192.168.8.15 |
K8s集群master |
|
|
2 |
Worker01 |
192.168.8.16 |
K8s集群worker01 |
|
|
3 |
Worker02 |
192.168.8.17 |
K8s集群worker02 |
|
|
4 |
公共服务 (public-server) |
192.168.8.18 |
Dns Nginx Harbor nfs |
域名1： dns.test.com 域名2： yaml.test.com 域名3： harbor.test.com 域名4： nfs.test.com |
2、项目拓扑
二、部署公共服务
1、部署DNS域名解析
[root@localhost ~]# hostnamectl set-hostname public-server
[root@localhost ~]# bash
[root@public-server ~]# systemctl disable firewalld;systemctl stop firewalld
[root@public-server~]#sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@public-server ~]# setenforce 0
[root@public-server ~]# yum -y install bind bind-chroot
[root@public-server ~]# vim /etc/named.conf
配置：
options {
listen-on port 53 { 127.0.0.1;any; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
recursing-file "/var/named/data/named.recursing";
secroots-file "/var/named/data/named.secroots";
allow-query { localhost;any; };
保存退出
[root@public-server ~]# vim /etc/named.rfc1912.zones
配置：
最后一行添加
zone "test.com" IN {
type master;
file "test.com";
allow-update { none; };
};
[root@public-server ~]# cd /var/named/
[root@public-server named]# ls
chroot data dynamic named.ca named.empty named.localhost named.loopback slaves
[root@public-server named]# cp -p named.localhost test.com
[root@public-server named]# vim test.com
修改：
$TTL 1D
@ IN SOA test.com. admin.test.com. (
0 ; serial
1D ; refresh
1H ; retry
1W ; expire
3H ) ; minimum
NS dns.test.com.
dns A 192.168.8.18
yaml A 192.168.8.18
harbor A 192.168.8.18
nfs A 192.168.8.18
保存
启动dns服务
[root@public-server named]# systemctl enable named
[root@public-server named]# systemctl start named
所有节点修改dns指定dns服务器
[root@所有节点r ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
DNS1=192.168.8.18
[root@所有节点~]# systemctl restart network
安装域名解析工具
[root@public-server named]# yum -y install bind-utils
域名解析 DNS解析验证
[root@public-server named]# nslookup dns.test.com
[root@public-server named]# nslookup harbor.test.com
[root@public-server named]# nslookup yaml.test.com
[root@public-server named]# nslookup nfs.test.com
2、YAML资源清单文件托管服务nginx配置
Yum安装nginx
[root@public-server ~]# yum -y install nginx
修改nginx配置文件
[root@public-server ~]# vim /etc/nginx/nginx.conf
修改
server {
listen 81;
listen [::]:81;
server_name _;
root /usr/share/nginx/html;
autoindex on;
index index;
启动nginx并设置开机自动启动
[root@public-server ~]# systemctl enable nginx
[root@public-server ~]# systemctl restart nginx
测试服务可用性
3、安装镜像仓库harbor
安装docker-ce
托包本地安装
[root@public-server ~]# cd docker-26/
[root@public-server docker-26]# yum -y localinstall *.rpm
以下操作不执行
安装docker-ce
[root@public-server ~]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
[root@public-server ~]# yum -y install docker-ce
注意：yum直接安装会出现Error downloading packages: docker-compose-plugin-2.27.1-1.el7.x86_64: [Errno 256] No more mirrors to try.
启动docker
[root@public-server ~]# systemctl enable docker
[root@public-server ~]# systemctl start docker
查看docker版本
[root@public-server ~]# docker version
安装docker-compose
托包docker-compose和harbor
[root@public-server~]#mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@public-server ~]# chmod +x /usr/local/bin/docker-compose
[root@public-server ~]# docker-compose -v
将harbor包解压到/usr/local/下
[root@public-server ~]# tar xf harbor-offline-installer-v2.3.5.tgz -C /usr/local/
[root@public-server ~]# cd /usr/local/harbor/
[root@public-server harbor]# cp harbor.yml.tmpl harbor.yml
[root@public-server harbor]# vim harbor.yml
修改：
注释https模块
hostname: harbor.test.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
#https:
# https port for harbor, default is 443
#port: 443
# The path of cert and key files for nginx
#certificate: /your/certificate/path
#private_key: /your/private/key/path
保存退出
导入镜像
[root@public-server harbor]# docker load -i harbor.v2.3.5.tar.gz
[root@public-server harbor]# docker images
[root@public-server harbor]# ./prepare
[root@public-server harbor]# ./install.sh
查看容器是否启动
[root@public-server harbor]# docker ps -a
测试harbor可用性
所有节点
[root@所有节点 ~]# vim /etc/docker/daemon.json
"insecure-registries": ["http://harbor.test.com"],
重载docker
[root@所有节点 ~]# systemctl reload docker
登录harbor
[root@public-server harbor]# docker login -uadmin -pHarbor12345 harbor.test.com
4、配置nfs
所有节点安装nfs-utils
[root@所有节点 ~]# yum -y install nfs-utils
配置nfs
[root@public-server harbor]# mkdir /data/nfs
[root@public-server ~]# vim /etc/exports
添加
/data/nfs *(rw,sync,no_root_squash)
重启服务并设置开机自启
[root@public-server ~]# systemctl restart nfs
[root@public-server ~]# systemctl enable nfs
所有节点测试是否能挂载nfs
[root@public-server ~]# showmount -e nfs.test.com
[root@master01 ~]# showmount -e nfs.test.com
[root@worker01 ~]# showmount -e nfs.test.com
[root@worker02 ~]# showmount -e nfs.test.com
三、配置nfs动态供给storageclass
1、下载并创建storageclass存储类
[root@master01 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/class.yaml
如果没有vpn可以复制内容到这个yaml文件
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
[root@master01 ~]# mv class.yaml storageclass-nfs.yml
[root@master01 ~]# kubectl apply -f storageclass-nfs.yml
[root@master01 ~]# kubectl get storageclasses.storage.k8s.io
2、下载资源清单文件，创建rbac账号并授权
没有vpn可以复制内容跟上面一样的操作
[root@master01 ~]# wget https://raw.githubusercontent.com/kubernetes-sigs/nfs-subdir-external-provisioner/master/deploy/rbac.yaml
[root@master01 ~]# mv rbac.yaml storageclass-nfs-rbac.yaml
3、创建动态供给的deployment
通过中间件将访问账号与共享存储关联
[root@master01 ~]# vim deploy-nfs-client-provisioner.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccount: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: registry.cn-beijing.aliyuncs.com/pylixm/nfs-subdir-external-provisioner:v4.0.0
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: k8s-sigs.io/nfs-subdir-external-provisioner
- name: NFS_SERVER
value: 192.168.8.18
- name: NFS_PATH
value: /data/nfs
volumes:
- name: nfs-client-root
nfs:
server: 192.168.8.18
path: /data/nfs
在工作节点worker01和worker02导镜像
[root@worker01 ~]# docker load -i nfs-provisioner.tar
[root@worker02 ~]# docker load -i nfs-provisioner.tar
执行资源清单文件
[root@master01 ~]# kubectl apply -f deploy-nfs-client-provisioner.yml
如果出现ContainerCreating情况可以使用如下命令查看
[root@master01 ~]# kubectl describe pod nfs-client-provisioner-7b77cfb575-689bc
如果是calico故障执行如下命令
[root@master01 ~]# kubectl delete pod -n calico-system –all
四、部署metallb
1、修改kube-proxy工作模式
[root@master01 ~]# kubectl edit configmap kube-proxy -n kube-system
修改：
[root@master01 ~]# kubectl rollout restart daemonset -n kube-system kube-proxy
查看工作模式
[root@master01 ~]# kubectl describe configmaps -n kube-system kube-proxy |grep mode
2、部署metallb
复制下面镜像备份文件复制到所有节点
[root@master01 ~]# docker load -i metallb.tar
[root@worker01 ~]# docker load -i metallb.tar
[root@worker02 ~]# docker load -i metallb.tar
将metallb-native.yaml复制到master
运行pod
[root@master01 ~]# kubectl apply -f metallb-native.yaml
[root@master01 ~]# kubectl get pod -n metallb-system
3、ip地址池准备
[root@master01 ~]# vim ippool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ippool
namespace: metallb-system
spec:
addresses:
- 192.168.8.240-192.168.8.250
[root@master01 ~]# kubectl apply -f ippool.yaml
[root@master01 ~]# kubectl get ipaddresspools.metallb.io -n metallb-system
4、开启二层通告
[root@master01 ~]# vim L2.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: example
namespace: metallb-system
[root@master01 ~]# kubectl apply -f L2.yaml
五、部署ingress nginx代理
1、导入ingress.tar镜像到所有工作节点，将资源清单复制到master
[root@worker01 ~]# docker load -i ingress.tar
[root@worker02 ~]# docker load -i ingress.tar
[root@master01 ~]# kubectl apply -f deploy.yaml
[root@master01 ~]# kubectl get pod -n ingress-nginx
六、项目基础镜像准备
本次发布一个java项目，以war包方式发布，需要使用tomcat做为项目基础镜像,然后定制。
1、定制tomcat镜像（在公共服务器上做）
[root@public-server ~]# mkdir /root/tomcat
[root@public-server ~]# cd /root/tomcat/
[root@public-server tomcat]# vim Dockerfile
FROM centos:7
MAINTAINER "admin<admin@test.com>"
ENV VERSION=9.0.91
ENV JAVA_HOME=/usr/local/jdk
COPY ./apache-tomcat-${VERSION}.tar.gz /
RUN tar xf /apache-tomcat-${VERSION}.tar.gz
RUN mv /apache-tomcat-${VERSION} /usr/local/tomcat
RUN rm -rf /apache-tomcat-${VERSION}.tar.gz /usr/local/tomcat/webapps/*
RUN mkdir /usr/local/tomcat/webapps/ROOT
ADD ./jdk /usr/local/jdk
RUN echo "export TOMCAT_HOME=/usr/local/tomcat" >> /etc/profile
RUN echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile
RUN echo "export PATH=$TOMCAT_HOME/bin:$JAVA_HOME/bin:$PATH" >> /etc/profile
RUN echo "export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar" >> /etc/profile
RUN source /etc/profile
EXPOSE 8080
CMD ["/usr/local/tomcat/bin/catalina.sh","run"]
保存退出
可提前导入centos:7的镜像，然后将jdk解压移动到/root/tomcat
[root@public-server tomcat]# docker load -i centos7.tar
[root@public-server ~]# mkdir /root/tomcat
[root@public-server ~]# cd /root/tomcat/
[root@public-server tomcat]# vim Dockerfile
[root@public-server tomcat]# cd
[root@public-server ~]# tar xf jdk-8u191-linux-x64.tar.gz
[root@public-server ~]# mv jdk1.8.0_191/ tomcat/jdk
[root@public-server ~]# cd tomcat/
[root@public-server tomcat]# ls
apache-tomcat-9.0.91.tar.gz Dockerfile jdk
2、开启路由转发
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
[root@public-server tomcat]# sysctl -p
3、构建镜像并推送到harbor
[root@public-server tomcat]# docker build -t harbor.test.com/java-project/tomcat:9102 .
windows浏览器访问harbor服务器
并创建java-project项目
[root@public-server tomcat]# docker push harbor.test.com/java-project/tomcat:9102
将mysql镜像推送到harbor仓库
[root@public-server ~]# docker load -i mysql5.7.tar
[root@public-server ~]# docker tag mysql:5.7 harbor.test.com/java-project/mysql:5.7
[root@public-server ~]# docker push harbor.test.com/java-project/mysql:5.7
将busybox镜像推送到harbor仓库
[root@public-server ~]# docker load -i busybox.1.28.tar
[root@public-server ~]# docker tag busybox:1.28.4 harbor.test.com/java-project/busybox:1.28.4
[root@public-server ~]# docker push harbor.test.com/java-project/busybox:1.28.4
在harbor验证tomcat镜像是否上传
七、存储准备
[root@master01 ~]# kubectl get storageclass
[root@master01 ~]# kubectl get pod
八、项目容器镜像仓库及项目源码准备
1、项目源码准备
复制java源码包javaproject.tar.gz到当前位置，并解包
[root@public-server ~]# tar xf javaproject.tar.gz
[root@public-server ~]# cd /root/javaproject/
main 源代码目录
[root@public-server java-project]# cd /root/javaproject/project-source/pro-source/java-project
[root@public-server java-project]# vim Dockerfile
修改：
FROM harbor.test.com/java-project/tomcat:9102
LABEL maintainer "admin <admin@test.com>"
RUN rm -rf /usr/local/tomcat/webapps/*
ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
[root@public-server java-project]# cat /root/javaproject/project-source/pro-source/java-
[root@public-server java-project]# cat /root/javaproject/project-source/pro-source/java-project/src/main/resources/application.yml
2、项目编译环境准备 jdk & maven
用于对java项目代码进行编译打包代码
下载maven，可以托包
[root@public-server java-project]# cd
安装maven
[root@public-server ~]# tar xf jdk-8u191-linux-x64.tar.gz
[root@public-server ~]# mv jdk1.8.0_191/ /usr/local/jdk
[root@public-server ~]# tar xf apache-maven-3.8.8-bin.tar.gz
[root@public-server ~]# mv apache-maven-3.8.8 /usr/local/maven
[root@public-server ~]# vim /etc/profile.d/maven.sh
添加
export JAVA_HOME=/usr/local/jdk
export MAVEN_HOME=/usr/local/maven
export PATH=${MAVEN_HOME}/bin:${JAVA_HOME}/bin:$PATH
[root@public-server ~]# source /etc/profile
九、项目部署
- 本项目部署思路
- 数据库 MySQL
- Web应用 Tomcat
- 有状态应用与无状态应用部署规划
Mysql
-
-
- statefulset控制器
- headless service
- pv,pvc 存储资源动态供给
-
tomcat（java应用）
-
-
- deployment控制器
- ClusterIP Service
- Ingress 服务暴露实现集群外访问
-
2、项目资源清单文件准备
[root@public-server ~]# mkdir /usr/share/nginx/html/java_project
[root@public-server ~]# cd /usr/share/nginx/html/java_project/
1.namespace资源清单文件(用于实现项目隔离)
[root@public-server java_project]# vim 01_ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: javaproject
2. 部署java项目资源清单文件(用于部署java项目)
[root@public-server java_project]# vim 02_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-project
namespace: javaproject
spec:
replicas: 2
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
containers:
- name: tomcat
image: harbor.test.com/java-project/java-project:v1 #镜像
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
3. 项目服务（service）资源清单文件(用于创建service)
[root@public-server java_project]# vim 03_service.yaml
apiVersion: v1
kind: Service
metadata:
name: java-project
namespace: javaproject
spec:
selector:
project: www
app: java-demo
ports:
- name: web
port: 80
targetPort: 8080
4.Ingress对象资源清单文件(需要提前部署 ingress控制器)
[root@public-server java_project]# vim 04_ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-project
namespace: javaproject
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
ingressClassName: nginx
rules:
- host: javaweb.test.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: java-project
port:
number: 80
5.mysql部署资源清单文件
[root@public-server java_project]# vim 05_mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: javaproject
spec:
ports:
- port: 3306
name: mysql
clusterIP: None
selector:
app: mysql-public
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
namespace: javaproject
spec:
selector:
matchLabels:
app: mysql-public
serviceName: "mysql"
template:
metadata:
labels:
app: mysql-public
spec:
containers:
- name: mysql
image: harbor.test.com/java-project/mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
- name: MYSQL_DATABASE
value: test
ports:
- containerPort: 3306
volumeMounts:
- mountPath: "/var/lib/mysql"
name: nfsdata
volumeClaimTemplates:
- metadata:
name: nfsdata
spec:
accessModes: ["ReadWriteMany"]
storageClassName: "nfs-client"
resources:
requests:
storage: 5Gi
6. mysql访问测试pod
[root@public-server java_project]# vim 06_create_pod_busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox-pod
spec:
containers:
- name: busybox-container
image: harbor.test.com/java-project/busybox:1.28.4 #此镜像nslookup及ping命令都没有问题，不要下载最新版本和1.31。
imagePullPolicy: IfNotPresent
command:
- sleep
- "3600"
restartPolicy: Always
查看资源清单
十、项目数据库 Mysql部署
1、创建命名空间
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/01_ns.yaml
2、部署数据库
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/05_mysql.yaml
查看持久化存储
3、导入数据库
[root@public-server java_project]# scp /root/javaproject/project-source/db/test.sql 192.168.8.15:/root
[root@master01 ~]# kubectl cp test.sql db-0:/ -n javaproject
[root@master01 ~]# kubectl exec -it db-0 -n javaproject -- bash
root@db-0:/# ls
bin boot dev docker-entrypoint-initdb.d entrypoint.sh etc home lib lib64 media mnt opt proc root run sbin srv sys test.sql tmp usr var
root@db-0:/# mysql -uroot -p123456
mysql> show databases;
mysql> use test
mysql> source /test.sql
mysql> show tables;
mysql> desc user;
4、验证数据库可用性
[root@master01~]#kubectl apply -f http://yaml.test.com:81/java_project/06_create_pod_busybox.yaml
[root@master01 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox-pod 1/1 Running 0 8s
nfs-client-provisioner-7b77cfb575-689bc 1/1 Running 0 101m
[root@master01 ~]# kubectl exec -it busybox-pod -- sh
/ # nslookup db-0.mysql.javaproject
/ # nslookup db-0.mysql.javaproject.svc.cluster.local.
十一、项目源码编译打包
1、修改项目连接数据库地址
[root@public-server ~]# cd javaproject/project-source/pro-source/java-project/
[root@public-server java-project]# pwd
/root/javaproject/project-source/pro-source/java-project
[root@public-server java-project]# ls
Dockerfile LICENSE pom.xml README.md src
[root@public-server java-project]# cd src/
[root@public-server src]# cd main/
[root@public-server main]# cd resources/
[root@public-server resources]# vim application.yml （不用修改）
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://db-0.mysql.javaproject:3306/test?characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
2、使用maven编译项目源码
[root@public-server java-project]# pwd
/root/javaproject/project-source/pro-source/java-project
[root@public-server java-project]# mvn clean package
过程慢长需等待
[root@public-server java-project]# ls
Dockerfile LICENSE pom.xml README.md src target
[root@public-server java-project]# ls target/
[root@public-server java-project]# docker build -t harbor.test.com/java-project/java-project:v1 .
[root@public-server java-project]# docker login harbor.test.com
Login Succeeded
[root@public-server java-project]# docker push harbor.test.com/java-project/java-project:v1
登录web查看
十二、部署项目
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/02_deployment.yaml
[root@master01 ~]# kubectl get pod -n javaproject
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/03_service.yaml
[root@master01 ~]# kubectl apply -f http://yaml.test.com:81/java_project/04_ingress.yaml
ingress.networking.k8s.io/java-project created
[root@master01 ~]# kubectl get ingress -n javaproject
[root@master01 ~]# kubectl get svc -n ingress-nginx
[root@master01 ~]# kubectl get ingress -n javaproject
十三、访问测试
windows浏览器访问：javaweb.test.com
需要设置hosts文件
C:\Windows\System32\drivers\etc\hosts
更多推荐
15
0- 0
已为社区贡献4条内容
所有评论(0)