部署java项目进k8s

java迁移k8s

weixin_45112997

1428人浏览 · 2022-06-17 17:14:24

weixin_45112997 · 2022-06-17 17:14:24 发布

1、确定应用在各个环境配置文件的值

依赖的中间件地址
- 数据库相关的：地址/用户名/密码/端口/DB
- Redis相关的：地址/用户名/密码/端口/DB
- kafka相关的：地址/用户名/密码/端口/Topic
依赖的其他应用访问地址
- (推荐)k8s内部访问地址：应用service名字.命名空间.svc
- 外网域名
- (不推荐)需映射hosts的域名
- (不推荐)IP地址加端口
日志组件

2、应用需要的持久化volume挂载路径

询问开发，应用有哪些需要持久化的目录和需要与其他应用程序共享的持久化目录

日志存储目录 /var/www/storage/log

3、应用依赖的系统命令

询问开发人员应用中是否有业务代码调用了系统命令，防止Docker基础镜像中没有安装

4、应用是否添加了健康检查Endpoint

为了在k8s中通过探针监测应用的存活状态，需要开发人员在应用代码中添加/health的Endpoint URL的代码。

5、应用Session是否已改造使用Redis

为了让应用能在k8s进行多实例部署、达到高可用。需要解决多实例间的session问题。需要开发人员将session存到Redis中。

确定存到Redis中的key的前缀唯一（CACHE_PREFIX=应用名_）
相关的配置
k8s configmap env

BROADCAST_DRIVER=log
CACHE_DRIVER=redis
SESSION_DRIVER=redis
SESSION_LIFETIME=120
QUEUE_DRIVER=sync
CACHE_PREFIX=应用名_

REDIS_HOST=redis地址
REDIS_PASSWORD=*****
REDIS_PORT=6379

6、访问域名

应用是否需要外网访问域名
域名对应的IP地址是否被其他地方限制

6.1、镜像制作

基础镜像

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

ENV JAVA_HOME=/opt/jdk-11.0.14\
    CLASSPATH=.:/opt/jdk-11.0.14/lib \
    LANG=zh_CN.UTF-8 \
    LANGUAGE=zh_CN:zh:en_US:en \
    TZ=Asia/Shanghai
ENV PATH=$PATH:/opt/jdk-11.0.14/bin
RUN sed -i "s/archive.ubuntu.com/mirrors.ustc.edu.cn/g" /etc/apt/sources.list && \
    sed -i "s/security.ubuntu.com/mirrors.ustc.edu.cn/g" /etc/apt/sources.list && \
    apt-get update &&  apt-get -y install --no-install-recommends \
    tzdata \
    language-pack-zh-hans \
    wget \
    dumb-init && \
    wget -q jdk-11.0.14_linux-x64_bin.tar.gz -P /opt/ && \
    cd /opt/ && tar xf jdk-11.0.14_linux-x64_bin.tar.gz && \
    rm -f jdk-11.0.14_linux-x64_bin.tar.gz && \
    cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone && \
    echo "en_US.UTF-8 UTF-8" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN.UTF-8 UTF-8" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN.GBK GBK" >> /var/lib/locales/supported.d/local && \
    echo "zh_CN GB2312" >> /var/lib/locales/supported.d/local && \
    locale-gen

ENTRYPOINT  ["dumb-init", "--"]

maven-Dockerfile

FROM **/java11-ubuntu2004:v1

LABEL Author=""


ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Asia/Shanghai\
    M2_HOME=/opt/apache-maven-3.8.3
ENV PATH=$PATH:${M2_HOME}/bin

USER root

COPY settings.xml /root/.m2/settings.xml
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    curl \
    git \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common && \
    curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg  | apt-key add - && \
    add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu focal stable" && \
    apt update && \
    apt install -y --no-install-recommends docker-ce-cli && \
    wget http://**/apache-maven-3.8.3-bin.tar.gz && \
    tar xf apache-maven-3.8.3-bin.tar.gz && \
    mv apache-maven-3.8.3 /opt/apache-maven-3.8.3 && \
    rm -f apache-maven-3.8.3-bin.tar.gz

6.2、makefile

IMAGE_BASE_PUSH = **/zz-med
IMAGE_NAME = java11-maven-builder
IMAGE_VERSION = v1
all: build push
build:
		docker build --rm -f Dockerfile -t ${IMAGE_BASE_PUSH}/${IMAGE_NAME}:${IMAGE_VERSION} .
push:
		docker push ${IMAGE_BASE_PUSH}/${IMAGE_NAME}:${IMAGE_VERSION}

6.3、.gitlab-ci.yml

image: **/java11-maven-builder:v1

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_DAEMON: |
    {
      "registry-mirrors": ["阿里云镜像加速地址"],
      "insecure-registries": ["harbor域名地址"]
    }

services:
  - name: docker:18.09-dind
    command:
      - /bin/sh
      - -c
      - |
        mkdir -p /etc/docker || exit
        echo "$DOCKER_DAEMON" > /etc/docker/daemon.json || exit
        dockerd-entrypoint.sh || exit
stages:
  - scan-code
  - build
  - deploy

scan-code:
  stage: scan-code
  script:
    - mvn package sonar:sonar 
      -Dsonar.host.url=http://sonarqube地址 
      -Dsonar.login=$SONARSCANNER_TOKEN 
      -Dsonar.qualitygate.wait=true 
      -Dsonar.qualitygate.timeout=500 
      -Dsonar.projectName=$CI_PROJECT_NAME 
      -Dsonar.projectKey=$CI_PROJECT_NAME 
      -Dsonar.projectVersion=$CI_COMMIT_SHA 
      -Dsonar.gitlab.project_id=$CI_PROJECT_ID 
      -Dsonar.gitlab.commit_sha=$CI_COMMIT_SHORT_SHA 
      -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME 
      -Dsonar.links.scm=$CI_PROJECT_URL 
      -Dsonar.links.ci=$CI_PIPELINE_URL 
      -Dsonar.sources=./src
      -Dsonar.test.inclusions=./*Test*/**
      -Dsonar.exclusions=./*Test*/**
      -Dsonar.java.binaries=target
      -DskipTests=true
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'release'
  tags:
    - gitlab-runner注册别名

#####################################################################
###                 构建阶段脚本                                     ##
#####################################################################
.k8s_build: &k8s_build
  - export RELEASE_NUM="$(date +%Y%m%d)-$(echo $CI_COMMIT_SHA | cut -c1-8)"
  - export IMAGE_NAME="hub.zz-med-stg.com/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"
  - mvn package -DskipTests=true
  - wget -q -nv -np -r -l=1 -nd --http-user=gitlab-ci --http-password=$APP_K8S_CONFIG_CENTER -R "index.html*" -P docker http://***/$CI_PROJECT_NAME/$APPENV/
  - docker login harbor地址 -u gitlab-ci -p $HARBOR_TOKEN
  - docker build -t "***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM" . -f docker/Dockerfile
  - docker push "***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"

###################################################################
###                 部署阶段脚本                                     ##
#####################################################################
.k8s_deploy: &k8s_deploy
  - export RELEASE_NUM="$(date +%Y%m%d)-$(echo $CI_COMMIT_SHA | cut -c1-8)"
  - export IMAGE_NAME="***/$CI_PROJECT_NAME/$APPENV:$RELEASE_NUM"
  - sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories  && apk update && apk add --no-cache wget
  - wget -q -nv -np -r -l=1 -nd --http-user=gitlab-ci --http-password=$APP_K8S_CONFIG_CENTER -R "index.html*" -P docker http://***/$CI_PROJECT_NAME/$APPENV/
  - chmod +x docker/config.sh && cd docker && ./config.sh ./k8s-application.tpl.yaml > k8s-application.yaml
  - cat k8s-application.yaml
  - kubectl apply -f k8s-application.yaml
  - kubectl -n $NAMESPACE rollout status --timeout=260s deployment/$CI_PROJECT_NAME || exit_code=$?
  - |
    if [ $exit_code -ne 0 ];then
      apk add curl jq --no-cache > /dev/null;
      ROLLBACK_ID=$(kubectl -n $NAMESPACE rollout undo deployment/$CI_PROJECT_NAME -ojson | jq -r '.status.observedGeneration');
      curl -s https://oapi.dingtalk.com/robot/send?access_token="$PIPELINE_DINGDING_ROBOT_TOKEN" -H 'Content-Type: application/json' -d '{"msgtype": "markdown","markdown": {"title": "Gitlab流水线部署失败","text": "['$CI_PROJECT_NAME']('$CI_PROJECT_URL'/-/tree/'$CI_BUILD_REF_NAME')的'$APPENV'环境第['$CI_PIPELINE_ID']('$CI_PIPELINE_URL')号流水线'$CI_JOB_STAGE'阶段失败，已回滚至最近一个稳定版本'$ROLLBACK_ID'，请检查相关错误！"},"at": {"isAtAll": true}}' > /dev/null;
      exit 1;
    fi

##############################################################################
#              Test environment build and deploy CI/CD                       #
##############################################################################
build-test:
  stage: build
  variables:
    APPENV: test
  script: *k8s_build
  only:
    - develop
  tags:
    - runner注册别名

deploy-test:
  stage: deploy
  image: ***/kubectl-stg镜像名称
  variables:
    NAMESPACE: test
    APPENV: test
    POD_NUM: 1
  script: *k8s_deploy
  only:
    - develop
  tags:
    - runner注册别名

##############################################################################
#              Stg environment build and deploy CI/CD                        #
##############################################################################
build-stg:
  stage: build
  variables:
    APPENV: stg
  script: *k8s_build
  only:
    - release
  tags:
    - runner注册别名

deploy-stg:
  stage: deploy
  image: ***/kubectl-stg:镜像名称
  variables:
    APPENV: stg
    NAMESPACE: stg
    POD_NUM: 1
  script: *k8s_deploy
  only:
    - release
  tags:
    - runner注册别名

##############################################################################
#              Production environment build and deploy CI/CD                 #
##############################################################################
build-prd:
  stage: build
  variables:
    APPENV: prod
  retry:
    max: 2
    when:
      - always
  script: *k8s_build
  only:
    - tags
  except:
    - release
    - develpo
  tags:
    - runner注册别名
  
deploy-prd:
  stage: deploy
  image: **/kubectl-prod:镜像名称
  variables:
    APPENV: prod
    NAMESPACE: prd
    POD_NUM: 2
  script: *k8s_deploy
  only:
    - tags
  except:
    - release
    - develop
  when: manual
  tags:
    - runner注册别名

6.4、deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: $CI_PROJECT_NAME
  namespace: $NAMESPACE
  labels:
    app: $CI_PROJECT_NAME
  annotations:
    kubernetes.io/change-cause: $IMAGE_NAME
spec:
  selector:
    matchLabels:
      app: $CI_PROJECT_NAME
  replicas: $POD_NUM
  template:
    metadata:
      labels:
        app: $CI_PROJECT_NAME
    spec:
      volumes:
        - name: log-local-storage
          hostPath:
            path: /log/app/$APPENV
      imagePullSecrets:
        - name: zzmed-registry
      containers:
        - name: app
          image: $IMAGE_NAME
          imagePullPolicy: Always
          args:
            - 'sh'
            - '-c'
            - 'java -jar -Duser.timezone=Asia/Shanghai -Xms1024M -Xmx1024M /application/app.jar'
          env:
            - name: K8s_POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: IDC
              value: $APPENV
            - name: APOLLO_CONFIGSERVICE
              value: http://apollo-service:8080
            - name: APP_ID
              value: $CI_PROJECT_NAME
            - name: APOLLO_NAMESPACE
              value: application
          volumeMounts:
            - name: log-local-storage
              mountPath: /logs
              subPath: $CI_PROJECT_NAME
          ports:
            - containerPort: 8080
              protocol: TCP
              name: app
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 60
            successThreshold: 1
            tcpSocket:
              port: 8080
            timeoutSeconds: 2
          readinessProbe:
            failureThreshold: 3
            periodSeconds: 60
            successThreshold: 1
            tcpSocket:
              port: 8080
            timeoutSeconds: 2
          resources:
            limits:
              cpu: "1000m"
              memory: 2000Mi
            requests:
              cpu: "100m"
              memory: 100Mi

6.5、应用Dockerfile

FROM **/java11-ubuntu2004:v1
COPY xx/target/*.jar /application/app.jar

6.6、config.sh

#!/bin/sh
if [ ! -n "$1" ];then
    echo "must input file name"
    exit 1
else
  cat $1 |
    sed 's/\$NAMESPACE'"/$NAMESPACE/g" |
    sed 's~\$IMAGE_NAME'"~$IMAGE_NAME~g" |
    sed 's~\$APPENV'"~$APPENV~g" |
    sed 's~\$POD_NUM'"~$POD_NUM~g" |
    sed 's~\$CI_PROJECT_NAME'"~$CI_PROJECT_NAME~g" |
    cat -
fi

exit 0

6.7、svc、ing文件

---
apiVersion: v1
kind: Service
metadata:
  name: ***-svc
  namespace: prd
spec:
  selector:
    app: ***
  ports:
    - name: app
      protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ***-ing
  namespace: prd
spec:
  rules:
  - host: ***.com
    http:
      paths:
      - path: /
        backend:
          serviceName: ***-svc
          servicePort: 80

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub