大数据-146 Apache Kudu 安装运行 Dockerfile 模拟集群 启动测试
本节对 Apache Kudu 进行部署,通过Docker Compose 配置文件,用于部署 Apache Kudu 集群。Apache Kudu 是一个用于快速分析和实时数据处理的分布式列式存储系统,常与 Apache Hadoop 生态系统中的其他组件(如 Apache Impala、Apache Spark)一起使用。
·
点一下关注吧!!!非常感谢!!持续更新!!!
目前已经更新到了:
- Hadoop(已更完)
- HDFS(已更完)
- MapReduce(已更完)
- Hive(已更完)
- Flume(已更完)
- Sqoop(已更完)
- Zookeeper(已更完)
- HBase(已更完)
- Redis (已更完)
- Kafka(已更完)
- Spark(已更完)
- Flink(已更完)
- ClickHouse(已更完)
- Kudu(正在更新…)
章节内容
上节我们完成了如下的内容:
- Kudu 的基本介绍
- Kudu 的架构概述
安装准备
硬件准备
- 一台或者多台跑kudu-master。建议一个Master(无容错机制)、三个Master(允许一个节点出错)、五个Master(允许两个节点出错)
- 一台或者多台机器跑kudu-server。当需要使用副本,至少需要三个节点运行kudu-server服务。
操作系统
- Ubuntu云服务器(我的环境是三台小云服务器)
- 内核和文件系统支持 Hole Punching 选项。
- NTP 服务
- xfs or ext4 formatterd drivers
存储要求
- 尽量使用固态存储,显著提供kudu性能
环境说明
- 如果你使用的是CDH,需要ClouderaManager 5.4.3 及以上的版本。
- 三台云服务器,h121启动Master,h122和123启动TabletServer。
Dockerfile
https://github.com/apache/kudu/blob/master/docker/quickstart.yml
官方的配置文件为:
- 3个Master
- 4个TabletServer
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
version: "3"
services:
kudu-master-1:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
ports:
- "7051:7051"
- "8051:8051"
command: ["master"]
volumes:
- kudu-master-1:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
# TODO: Use `host.docker.internal` instead of KUDU_QUICKSTART_IP when it
# works on Linux (https://github.com/docker/for-linux/issues/264)
- >
MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master
--rpc_bind_addresses=0.0.0.0:7051
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP:?Please set KUDU_QUICKSTART_IP environment variable}:7051
--webserver_port=8051
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8051
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-master-2:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
ports:
- "7151:7151"
- "8151:8151"
command: ["master"]
volumes:
- kudu-master-2:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master
--rpc_bind_addresses=0.0.0.0:7151
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7151
--webserver_port=8151
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8151
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-master-3:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
ports:
- "7251:7251"
- "8251:8251"
command: ["master"]
volumes:
- kudu-master-3:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master
--rpc_bind_addresses=0.0.0.0:7251
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7251
--webserver_port=8251
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8251
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-tserver-1:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
- kudu-master-2
- kudu-master-3
ports:
- "7050:7050"
- "8050:8050"
command: ["tserver"]
volumes:
- kudu-tserver-1:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7050
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7050
--webserver_port=8050
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8050
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-tserver-2:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
- kudu-master-2
- kudu-master-3
ports:
- "7150:7150"
- "8150:8150"
command: ["tserver"]
volumes:
- kudu-tserver-2:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7150
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7150
--webserver_port=8150
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8150
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-tserver-3:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
- kudu-master-2
- kudu-master-3
ports:
- "7250:7250"
- "8250:8250"
command: ["tserver"]
volumes:
- kudu-tserver-3:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7250
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7250
--webserver_port=8250
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8250
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-tserver-4:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
- kudu-master-2
- kudu-master-3
ports:
- "7350:7350"
- "8350:8350"
command: ["tserver"]
volumes:
- kudu-tserver-4:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7350
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7350
--webserver_port=8350
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8350
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
kudu-tserver-5:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
depends_on:
- kudu-master-1
- kudu-master-2
- kudu-master-3
ports:
- "7450:7450"
- "8450:8450"
command: ["tserver"]
volumes:
- kudu-tserver-5:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver
--rpc_bind_addresses=0.0.0.0:7450
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7450
--webserver_port=8450
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8450
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
volumes:
kudu-master-1:
kudu-master-2:
kudu-master-3:
kudu-tserver-1:
kudu-tserver-2:
kudu-tserver-3:
kudu-tserver-4:
kudu-tserver-5:
内容解释
- version: “3”:指定使用 Docker Compose 文件的版本。版本 3 是当前常用的版本,支持多种高级功能,适用于 Docker 1.13.0 及更高版本。
- services:定义了多个服务,每个服务对应一个 Docker 容器。在本配置中,定义了多个 Kudu Master 和 Kudu Tablet Server(简称 TServer)。
- volumes:定义了持久化存储卷,用于存储各个服务的数据,确保即使容器重新创建或删除,数据仍然保留。
3 个 Kudu Master 节点:
- kudu-master-1
- kudu-master-2
- kudu-master-3
5 个 Kudu Tablet Server 节点:
- kudu-tserver-1
- kudu-tserver-2
- kudu-tserver-3
- kudu-tserver-4
- kudu-tserver-5
Master
每个 Master 节点的配置基本相同,只是端口和持久化卷不同。以下以 kudu-master-1 为例进行解释:
kudu-master-1:
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}
ports:
- "7051:7051"
- "8051:8051"
command: ["master"]
volumes:
- kudu-master-1:/var/lib/kudu
environment:
- KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
- >
MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master
--rpc_bind_addresses=0.0.0.0:7051
--rpc_advertised_addresses=${KUDU_QUICKSTART_IP:?Please set KUDU_QUICKSTART_IP environment variable}:7051
--webserver_port=8051
--webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8051
--webserver_doc_root=/opt/kudu/www
--stderrthreshold=0
--use_hybrid_clock=false
--unlock_unsafe_flags=true
- udu-master-1:服务名称,标识第一个 Kudu Master 节点。
- image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}:指定使用的 Docker 镜像。apache/kudu 是官方 Kudu 镜像。
- ${KUDU_QUICKSTART_VERSION:-latest}:使用环境变量 KUDU_QUICKSTART_VERSION 指定版本,如果未设置,则默认使用 latest。
- ports:“7051:7051”:将主机的 7051 端口映射到容器的 7051 端口,用于 Kudu Master 的 RPC 通信。“8051:8051”:将主机的 8051 端口映射到容器的 8051 端口,用于 Kudu Master 的 Web 界面。
- command: [“master”]:指定容器启动时运行的命令,这里启动 Kudu Master 进程。
- volumes:kudu-master-1:/var/lib/kudu:将 Docker 卷 kudu-master-1 挂载到容器内的 /var/lib/kudu 目录,用于持久化存储数据。
- KUDU_MASTERS:定义了集群中所有 Master 节点的地址,格式为 hostname:port,以逗号分隔。
启动服务
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1)
docker-compose up -d
访问服务
http://localhost:8051/
这样我们的服务就顺利的启动了:
更多推荐
已为社区贡献7条内容
所有评论(0)