k8s 部署canal admin及server单机服务

主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费。部署清单部署步骤a. 初始化数据库及创建用户b. 编写 K8s yaml文件mysql执行以下sql脚本，创建库 'canal_manager' 用来存放canal-admin的数据，server、instance配置等数据2.2 canal-admin k8s yaml部署文件创建部署 deployment.yaml，资源清

lv2

2011人浏览 · 2023-01-04 11:25:43

lv2 · 2023-01-04 11:25:43 发布

1. 前言

2. 部署canal-admin

2.1 数据库初始化

2.2 canal-admin k8s yaml部署文件

3. 部署canal-server单机

1. 前言

canal官方文档：https://github.com/alibaba/canal/wiki

主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费。

部署清单

软件	版本	备注
canal-admin	v1.1.6	镜像：canal/canal-admin:v1.1.6
canal-server	v1.1.6	镜像：canal/canal-server:v1.1.6,单机部署
mysql	5.7.19

部署步骤
a. 初始化数据库及创建用户
b. 编写 K8s yaml文件

2. 部署canal-admin

2.1 数据库初始化

mysql执行以下sql脚本，创建库 'canal_manager' 用来存放canal-admin的数据，server、instance配置等数据

# 初始化sql脚本
https://raw.githubusercontent.com/alibaba/canal/canal-1.1.6/admin/admin-web/src/main/resources/canal_manager.sql 

# 具体如下
CREATE DATABASE /*!32312 IF NOT EXISTS*/ `canal_manager` /*!40100 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin */;

USE `canal_manager`;

SET NAMES utf8;
SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------
-- Table structure for canal_adapter_config
-- ----------------------------
DROP TABLE IF EXISTS `canal_adapter_config`;
CREATE TABLE `canal_adapter_config` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `category` varchar(45) NOT NULL,
  `name` varchar(45) NOT NULL,
  `status` varchar(45) DEFAULT NULL,
  `content` text NOT NULL,
  `modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for canal_cluster
-- ----------------------------
DROP TABLE IF EXISTS `canal_cluster`;
CREATE TABLE `canal_cluster` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(63) NOT NULL,
  `zk_hosts` varchar(255) NOT NULL,
  `modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for canal_config
-- ----------------------------
DROP TABLE IF EXISTS `canal_config`;
CREATE TABLE `canal_config` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `cluster_id` bigint(20) DEFAULT NULL,
  `server_id` bigint(20) DEFAULT NULL,
  `name` varchar(45) NOT NULL,
  `status` varchar(45) DEFAULT NULL,
  `content` text NOT NULL,
  `content_md5` varchar(128) NOT NULL,
  `modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  UNIQUE KEY `sid_UNIQUE` (`server_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for canal_instance_config
-- ----------------------------
DROP TABLE IF EXISTS `canal_instance_config`;
CREATE TABLE `canal_instance_config` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `cluster_id` bigint(20) DEFAULT NULL,
  `server_id` bigint(20) DEFAULT NULL,
  `name` varchar(45) NOT NULL,
  `status` varchar(45) DEFAULT NULL,
  `content` text NOT NULL,
  `content_md5` varchar(128) DEFAULT NULL,
  `modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  UNIQUE KEY `name_UNIQUE` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for canal_node_server
-- ----------------------------
DROP TABLE IF EXISTS `canal_node_server`;
CREATE TABLE `canal_node_server` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `cluster_id` bigint(20) DEFAULT NULL,
  `name` varchar(63) NOT NULL,
  `ip` varchar(63) NOT NULL,
  `admin_port` int(11) DEFAULT NULL,
  `tcp_port` int(11) DEFAULT NULL,
  `metric_port` int(11) DEFAULT NULL,
  `status` varchar(45) DEFAULT NULL,
  `modified_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for canal_user
-- ----------------------------
DROP TABLE IF EXISTS `canal_user`;
CREATE TABLE `canal_user` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `username` varchar(31) NOT NULL,
  `password` varchar(128) NOT NULL,
  `name` varchar(31) NOT NULL,
  `roles` varchar(31) NOT NULL,
  `introduction` varchar(255) DEFAULT NULL,
  `avatar` varchar(255) DEFAULT NULL,
  `creation_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

SET FOREIGN_KEY_CHECKS = 1;

-- ----------------------------
-- Records of canal_user
-- ----------------------------
BEGIN;
INSERT INTO `canal_user` VALUES (1, 'admin', '6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9', 'Canal Manager', 'admin', NULL, NULL, '2019-07-14 00:05:28');
COMMIT;

SET FOREIGN_KEY_CHECKS = 1;

2.2 canal-admin k8s yaml部署文件

创建部署 deployment.yaml，资源清单包含：service，deployment，configmap，ingress。使用官方canal-admin镜像部署，可以通过环境变量的方式传递数据库配置（而不用修改canal-admin/conf/application.yml文件），因此，configmap存放了数据库等相关配置

canal-admin服务监听端口为：8089

canal-admin 部署文件

apiVersion: v1
kind: Service
metadata:
  name: canal-admin
spec:
  selector:
    app: canal-admin
  ports:
    - name: http
      protocol: TCP
      port: 8089
      targetPort: 8089 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: canal-admin
  labels:
    app: canal-admin
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - canal-admin
        topologyKey: "kubernetes.io/hostname"
  replicas: 1
  selector:
    matchLabels:
      app: canal-admin
  template:
    metadata:
      labels:
        app: canal-admin
    spec:
      restartPolicy: "Always"
      containers:
      - name: canal-admin
        image: canal/canal-admin:v1.1.6
        imagePullPolicy: "IfNotPresent"
        resources:
          limits:
            cpu: "1000m"
            memory: "3Gi"
          requests:
            cpu: "50m"
            memory: "2Gi"
        ports:
        - containerPort: 8089
          name: http
          protocol: TCP
        envFrom:
          - configMapRef:
              name: canal-admin-cm
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 5
          initialDelaySeconds: 10
          successThreshold: 1
          httpGet:
            port: 8089
            path: /
        livenessProbe:
          httpGet:
            port: 8089
            path: /
          initialDelaySeconds: 10
          periodSeconds: 10
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canal-admin-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: canal-admin.test.cn
    http:
      paths:
      - backend:
          service:
            name: canal-admin
            port:
              number: 8089
        path: /
        pathType: Prefix
---
# configmap
# kubectl create configmap game-config --from-file=docs/user-guide/configmap/kubectl
apiVersion: v1
kind: ConfigMap
metadata:
  name: canal-admin-cm
data:
  # replace your filename of properties configration and contents here.
  # canal-admin服务端口
  server.port: "8089"
  # canal-server注册到canal-admin时用的用户
  canal.adminUser: "admin"
  # canal-server注册到canal-admin时用的密码,注意这不是canal-admin 管理 web 的登录密码
  canal.adminPasswd: "123456"
  # 存放canal-admin数据的外部数据库配置
  spring.datasource.address: "192.168.1.221:3306"
  spring.datasource.database: "canal_manager"
  spring.datasource.username: "canal"
  spring.datasource.password: "canal"

部署

# kubectl部署
kubectl apply -f deployment.yaml
# 查看部署结果
kubectl get svc 
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
canal-admin             ClusterIP   10.43.215.112   <none>        8089/TCP                              4d21h
kubectl get pod -l app=canal-admin
NAME                           READY   STATUS    RESTARTS   AGE
canal-admin-795bb556d7-l2jgh   1/1     Running   0          23h
> kubectl get ingress -n tools
NAME                    CLASS   HOSTS                    ADDRESS        PORTS     AGE
canal-admin-ingress     nginx   canal-admin.test.cn     192.168.1.145   80      4d21h

部署完可以通过ingress配置的域名进行访问，如：http://canal-admin.test.cn，默认帐号密码：admin/123456

3. 部署canal-server单机

canal-server这里部署的是单机版，不需要sql初始化，直接部署即可，注意部署中的变量传递，canal-admin password是在2.2章节中的canal.adminPasswd的密文模式

密文模式

# 2.2配置的 canal.adminPasswd: "123456"，则明文密码为：123456，密文为
# 登录任一mysql，通过select查询语句 
 select password('123456');
 password('123456')
 *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9
# 密文为查询结果去掉*的内容，如下
6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9

canal-server 部署文件
服务端口11110，用于canal-admin通信，开启自动注册到canal-admin，注册IP为canal-server-headless.svc

apiVersion: v1
kind: Service
metadata:
  name: canal-server-headless
  labels:
    app: canal-server
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 11110
      name: http
      targetPort: 11110
    - name: tcp11112
      protocol: TCP
      port: 11112
      targetPort: 11112    
  selector:
    app: canal-server
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: canal-server
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - canal-server
        topologyKey: "kubernetes.io/hostname"
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: canal-server
  template:
    metadata:
      labels:
        app: canal-server
    spec:
      containers:
        - name: canal-server
          imagePullPolicy: Always
          image: canal/canal-server:v1.1.6
          resources:
            limits:
              memory: 3Gi
              cpu: 1000m
            requests:
              cpu: 50m
              memory: 2Gi
          ports:
            - containerPort: 11110
              name: http
              protocol: TCP  
            - containerPort: 11112
              name: tcp11112
              protocol: TCP  
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: SERVICE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: 'metadata.labels[''app'']'
            - name: STS_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: canal.register.ip
              value: canal-server-headless.svc
            - name: canal.admin.manager
              value: 'canal-admin:8089'
            - name: canal.admin.port
              value: '11110'
            - name: canal.admin.user
              value: 'admin'
            - name: canal.admin.passwd # 注意这里是2.2章节配置的canal-admin password密文
              value: '6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9'
            - name: canal.admin.register.auto
              value: 'true'
            - name: canal.admin.register.name
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name

部署

# kubectl部署
kubectl apply -f deployment.yaml
kubectl get svc 
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
canal-server-headless   ClusterIP   None            <none>        11110/TCP,11111/TCP,11112/TCP         23h
kubectl get pod -l app=canal-server
NAME             READY   STATUS    RESTARTS   AGE
canal-server-0   1/1     Running   0          18h

部署成功后，可以在canal-admin查看注册结果及运行状态

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub