flink on k8s 实践

1、Helm安装

wget https://get.helm.sh/helm-v3.11.2-linux-amd64.tar.gz

tar -zxvf helm-v3.11.2-linux-amd64.tar.gz

mv linux-amd64/helm /usr/local/bin/

helm安装好后，可以添加常用的helm源。

helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.3.1/

helm repo add stable http://mirror.azure.cn/kubernetes/charts

helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

2、安装cert-manager

下载 https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml

使用kubectl命令安装即可

kubectl apply -f cert-manager.yaml

 kubectl get all -n cert-manager

3、Flink Kubernetes Operator安装

 Flink Kubernetes Operator最简单直接的安装方式就是使用helm在线安装，命令如下：

helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-1.4.0/

helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator --namespace flink --create-namespace

helm -n flink ls -a

如果需要删除

helm -n flink delete flink-kubernetes-operator

kubectl get all -n flink -owide

helm list -n flink

 application 模式：

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment  # Flink集群在K8s的资源类型
metadata:
  name: basic-example  # 作业的名字
  namespace: flink        # 指定在flink命名空间下运行
spec:
  #image: flink:1.13.6      # Flink的镜像，改为使用1.13.6

   image: registry.cn-hangzhou.aliyuncs.com/cm_ns01/flink:1.14.6  
   flinkVersion: v1_14    # Flink的版本，与镜像版本保持一致
   flinkConfiguration:
     taskmanager.numberOfTaskSlots: "2"
   serviceAccount: flink
   jobManager:
     resource:
       memory: "1024m"
       cpu: 1
   taskManager:
     resource:
       memory: "1024m"
       cpu: 1
   job:
     jarURI: local:///opt/flink/flink-1.14.6/examples/streaming/StateMachineExample.jar  # Flink作业的启动类所在的Jar包路径
     parallelism: 2
     upgradeMode: stateless

kubectl apply -f basic.yaml 报错：

 kubectl delete -A ValidatingWebhookConfiguration flink-operator-flink-clusters-webhook-configuration

kubectl delete -A ValidatingWebhookConfiguration flink-operator-flink-webhook-configuration

kubectl apply -f basic.yaml

get all -n flink

kubectl get FlinkDeployment -n flink

停止作业

kubectl delete -f basic.yaml
kubectl delete FlinkDeployment basic-example -n flink

session 部署方式：

#Flink Session集群 源码请到 
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  namespace: flink
  name: session-deployment-only
spec:
  image: flink:1.14.6
  flinkVersion: v1_14
  imagePullPolicy: IfNotPresent   # 镜像拉取策略，本地没有则从仓库拉取
  ingress:   # ingress配置，用于访问flink web页面
    template: "flink.k8s.io"
    className: "nginx"
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: "/"
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "36"
  serviceAccount: flink
  jobManager:
    replicas: 1
    resource:
      memory: "1024m"
      cpu: 1
  taskManager:
    replicas: 1
    resource:
      memory: "1024m"
      cpu: 1

kubectl apply -f session-deployment-only.yaml

kubectl delete -f session-deployment-only.yaml

查看日志：

kubectl get pod -n flink | grep session

查看镜像，没有本地镜像仓库，需要添加

 cat /etc/docker/daemon.json
{
   "registry-mirrors": ["http://hub-mirror.c.163.com"],
   "insecure-registries": ["http://192.168.1.249:8086","https://192.168.1.249:16443"]
}
systemctl daemon-reload

systemctl restart docker

打tag

docker tag flink14:latest  192.168.1.249:8086/bigdata/flink14:v1_14 

docker push 192.168.1.249:8086/bigdata/flink14:v1_14

有权限异常报错，需要登录下

docker login 192.168.1.249:8086

admin / Harbor12345

 再次push 成功

pull下验证

docker pull 192.168.1.249:8086/bigdata/flink14:v1_14

也需要到搭建的node上去执行下 docker pull

 

 kubectl get all -n flink

删除重装操作：

kubectl delete -f session-deployment-only.yaml

4、安装ingress

下载

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.1/deploy/static/provider/cloud/deploy.yaml

mv deploy.yaml ingress-nginx-deploy.yaml

vim ingress-nginx-deploy.yaml

替换为国内的源 

将ingress-nginx-deploy.yaml文件中的镜像地址registry.k8s.io替换为 registry.lank8s.cn后再安装即可

kubectl apply -f ingress-nginx-deploy.yaml

kubectl get pods -n ingress-nginx

 kubectl get all -n ingress-nginx -owide

http://flink.k8s.io:32453/

与Dinky的配置