StreamPark集成k8s运行Flink实战
摘要:本文通过使用docker、k8s、streampark、flink完成了flink任务基于streampark平台的管理,资源方面集成k8s管理的实战
·
集群配置(Centos7)
node1(4vCpu,16G) | node2(4vCpu,16G) | node3(4vCpu,16G) |
docker | docker | docker |
k8s-master | k8s-node1 | k8s-node2 |
java-8 | ||
mysql-57 | ||
maven-3.9.6(可选) | ||
flink-1.16.3 | ||
streampark-2.1.3_2.12 |
Docker安装(三台机器)
#卸载自带docker
sudo yum remove docker*
#安装工具类
sudo yum install -y yum-utils
#配置docker的yum地址
sudo yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#安装docker
sudo yum install -y docker-ce docker-ce-cli containerd.io
# 启动&开机启动docker
systemctl enable docker --now
# docker加速配置
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
安装完成后可以使用docker ps 查看docker运行状态
docker ps
k8s安装
基本环境配置(三台机器):
确保每个机器使用内网ip互通
#设置每个机器自己的hostname
hostnamectl set-hostname xxx(更换为自己的主机名)
# 将 SELinux 设置为 permissive 模式(相当于将其禁用)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
#关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
#允许 iptables 检查桥接流量
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
安装kubelet、kubeadm、kubectl(三台机器)
#配置k8s的yum源地址
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#安装 kubelet,kubeadm,kubectl
sudo yum install -y kubelet-1.20.9 kubeadm-1.20.9 kubectl-1.20.9
#启动kubelet
sudo systemctl enable --now kubelet
#所有机器配置master域名
echo "<你的k8s-master主机ip> k8s-master" >> /etc/hosts
#ping一下检测连通性
ping k8s-master
初始化master节点(k8s-master机器)
kubeadm init \
--apiserver-advertise-address=<你的k8s-master主机ip> \
--control-plane-endpoint=k8s-master \
--image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
--kubernetes-version v1.20.9 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=192.168.0.0/16
记录打印信息并执行3-5行(k8s-master机器)
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
--discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
--discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39
安装Calico网络插件(注意版本问题:使用v3.20)(k8s-master机器)
curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O
kubectl apply -f calico.yaml
#检测安装情况(都在runing状态再向下进行)
kubectl get pod -A
加入worker节点(复制打印信息中的内容)(k8s-worker机器)
kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
--discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39
检测节点运行情况(k8s-master机器)
#都在readly状态证明安装完成
kubectl get node
java安装
上传jdk-8u401-linux-x64.tar.gz到/opt/software目录下
#解压
tar -zxvf jdk-8u401-linux-x64.tar.gz -C /opt/module
#创建环境变量
vim /etc/profile.d/myenv.sh
#设置JAVA_HOME
#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_401
export PATH=${JAVA_HOME}/bin:$PATH
#刷新生效
source /etc/profile.d/myenv.sh
#检验
java -version
mysql安装
#检查是否安装过MySQL
rpm -qa | grep mysql
#检查是否存在 mariadb 数据库(内置的MySQL数据库),有则强制删除
rpm -qa | grep mariadb
#强制删除
rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64
#强制删除
rpm -e --nodeps mariadb-5.5.68-1.el7.x86_64
#下载mysql源
curl -O https://repo.mysql.com//mysql57-community-release-el7-11.noarch.rpm
#安装MySQL源
yum localinstall mysql57-community-release-el7-11.noarch.rpm
#安装密钥
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
#安装mysql
yum install mysql-community-server
#启动mysql
systemctl start mysqld
#获取默认密码
grep 'temporary password' /var/log/mysqld.log
#登录mysql
mysql -uroot -p显示的默认密码
#修改密码策略
set global validate_password_policy=0;
set global validate_password_length=1;
#修改密码
ALTER USER 'root'@'localhost' IDENTIFIED BY '新密码';
#授予root用户远程访问权限,并刷新权限使生效(可选)
grant all privileges on *.* to 'root' @'%' identified by '新密码';
maven安装(用于依赖项下载,可选)
#下载安装包
curl -O https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
#解压
tar -zxvf apache-maven-3.9.6-bin.tar.gz -C /opt/module
flink安装
#下载安装包
curl -O https://archive.apache.org/dist/flink/flink-1.16.3/flink-1.16.3-bin-scala_2.12.tgz
#解压
tar -zxvf flink-1.16.3-bin-scala_2.12.tgz -C /opt/module
Streampark安装
官网手册:安装部署 | Apache StreamPark (incubating)
上传apache-streampark_2.12-2.1.3-incubating-bin.tar.gz到/opt/software目录下
#解压
tar -zxvf apache-streampark_2.12-2.1.3-incubating-bin.tar.gz -C /opt/module
#初始化表结构、登录mysql
source streampark安装位置/script/schema/mysql-schema.sql
source streampark安装位置/script/data/mysql-data.sql
#修改连接信息,进入到 conf 下,修改 conf/application.yml,找到 spring 这一项,找到 profiles.active 的配置,修改成mysql即可
spring:
profiles.active: mysql #[h2,pgsql,mysql]
#修改 config/application-mysql.yml
spring:
datasource:
username: root
password: 你的密码如果是纯数字需要加单引号'000000'
🤣#####!!!!!!上传mysql-connector-java到lib目录下!!!!!!!#######🤣
driver-class-name: com.mysql.jdbc.Driver # 请根据mysql-connector-java版本确定具体的路径,例如:使用5.x则此处的驱动名称应该是:com.mysql.jdbc.Driver
url: jdbc:mysql://localhost:3306/streampark?useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=false&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=GMT%2B8
#修改workspace
进入到 conf 下,修改 conf/application.yml,找到 streampark 这一项,找到 workspace 的配置,修改成一个用户有权限的目录
streampark:
# HADOOP_USER_NAME 如果是on yarn模式( yarn-prejob | yarn-application | yarn-session)则需要配置 hadoop-user-name
hadoop-user-name: hdfs
# 本地的工作空间,用于存放项目源码,构建的目录等.
workspace:
local: /opt/streampark_workspace #🤣手动建一下🤣
remote: hdfs:///streampark # support hdfs:///streampark/ 、 /streampark 、hdfs://host:ip/streampark/
#启动
startup.sh
#访问节点的10000端口即可进入(默认账号密码:admin / streampark)
集成k8s
官方手册:Flink K8s 集成支持 | Apache StreamPark (incubating)
#StreamPark 直接使用系统 ~/.kube/config 作为 Kubernetes 集群的连接凭证
#通过 StreamPark 所在机器的 kubectl 快速检查目标 Kubernetes 集群的连通性:
kubectl cluster-info
#Kubernetes RBAC 配置,假设使用 Flink Namespace 为 flink-dev 用户为default
kubectl create clusterrolebinding flink-role-binding-default --clusterrole=edit --serviceaccount=flink-dev:default
搭建docker私服
#pull下来registry镜像
docker pull registry:2.7.0
#创建帐号密码文件,写入帐号密码
mkdir -p /mydisk/docker/auth
cd /mydisk/docker/auth
echo "user:docker passwd:123456" >htpasswd
#加密帐号密码文件
docker run --entrypoint htpasswd registry:2.7.0 -Bbn docker 123456 > /mydisk/docker/auth/htpasswd
#启动镜像服务器
docker run -itd -p 5000:5000 --restart=always --name docker-registry -v /mydisk/docker/auth:/auth -v /mydisk/docker/registry:/var/lib/registry -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" registry:2.7.0
#修改本地docker私服配置
vim /etc/docker/daemon.json
#添加你的配置
{
"insecure-registries": ["私服的IP地址或者域名:端口号"]
}
#重启docker
systemctl restart docker.service
docker login 你私服ip或者域名:5000
#输入你的帐号密码
运行个任务
配置flink版本
sql任务
CREATE TABLE datagen_source (
name VARCHAR,
score BIGINT
) WITH (
'connector' = 'datagen'
);
CREATE TABLE datagen_sink (
name VARCHAR,
score BIGINT
) WITH (
'connector' = 'print'
);
INSERT INTO datagen_sink select * from datagen_source;
k8s对外服务类型ClusterIP or NodePort,否则无法与k8s准确交互
k8s pod 模板需要设置否则无法从私服拉取镜像
kubectl create secret generic <secret 名字> \
-n <namespace 名字> \
--from-file=.dockerconfigjson=<path/to/.docker/config.json> \
--type=kubernetes.io/dockerconfigjson
apiVersion: v1
kind: Pod
metadata:
name: pod-template
spec:
serviceAccount: default
containers:
- name: flink-main-container
image:
imagePullSecrets:
- name: <secret 名字>
提交-发布-运行即可
更多推荐
已为社区贡献1条内容
所有评论(0)