使用Podman搭建简单Hadoop集群
·
本文提供学习参考；集群用户密码请不要外泄！
一、环境
1.版本
VMware： 17
Ubuntu: 24
JDK：1.8.0_381
Zookeeper：3.8.0
Hadoop：3.3.4
Hive：3.1.3
Spark：3.3.4
Podman：4.9.3
2.节点
| 节点 | IP | 角色 | 运行服务 | 用户名 | 密码 |
| master | 10.99.0.10 | NameNode, SecondaryNameNode, ResourceManager, Spark Master, HiveServer2, MySQL | namenode, secondarynamenode, resourcemanager, master, hiveserver, mysqld | hadoop | hadoop |
| node1 | 10.99.0.11 | DataNode, NodeManager, Spark Worker, ZooKeeper | datanode, nodemanager, worker, zookeeper | ||
| node2 | 10.99.0.12 | DataNode, NodeManager, Spark Worker, ZooKeeper | datanode, nodemanager, worker, zookeeper | ||
| node3 | 10.99.0.13 | DataNode, NodeManager, Spark Worker, ZooKeeper | datanode, nodemanager, worker, zookeeper |
二、步骤与详细过程
（一）准备
1.在Ubuntu安装ssh与vim
1.1输入
sudo apt update1.2输入
sudo apt install openssh-server vim -y
2.宿主机配置静态IP
2.1输入
ip route获取网关
（下图网关192.168.146.2，每一个网关都不一样，要自己获取）
2.2输入
cd /etc/netplan输入
ls
2.3输入
sudo vim 50-cloud-init.yaml修改成以下内容，按下Esc输入:wq保存
（
ip route获取到网关为192.168.58.2）
network:
version: 2
ethernets:
ens33:
dhcp4: no
addresses:
- 192.168.58.200/24
routes:
- to: default
via: 192.168.58.2
nameservers:
addresses: [202.120.224.26]注意：
网关地址不一定都一样，需要通过ip route获得；
手动配置IP需要在一定范围内，例如，查到的网关是192.168.146.2，IP可以设定在192.168.146.1-192.168.146.254之间但不能与网关相同；
如果出现无法SSH连接，或无法连接互联网，查看网关是否改变，如改变需重新分配IP；
/etc/netplan里的文件不一定叫50-cloud-init.yaml，根据ls的结果进行vim修改
2.4重启网络配置
2.4.1输入
sudo netplan apply2.4.2输入
ip a show ens33
2.5测试连通网关
输入
ping 192.168.58.2 -c 3
2.6测试连通互联网
输入（以ping必应为例）
ping www.bing.com -c 3
3.在Windows系统操作
3.1Win+R输入cmd
3.2输入
ssh hadoop@<自己配置的IP>3.3弹出询问输入yes
3.4输入hadoop密码
（二）
1.安装Podman
1.1输入
sudo apt-get update
1.2输入
sudo apt-get install -y podman
1.3验证安装。输入
podman version
2.创建Podman网络
2.1输入
podman network create --subnet 10.99.0.0/16 --gateway 10.99.0.1 hadoop-cluster
2.2输入，查看是否创建成功
podman network inspect hadoop-cluster
3.创建持久化数据卷
3.1创建卷，输入
podman volume create hadoop-all-data3.2创建卷，输入
podman volume create hadoop-all-data-13.3创建卷，输入
podman volume create hadoop-all-data-23.4创建卷，输入
podman volume create hadoop-all-data-33.5输入
podman volume create mysql-data3.6输入
podman volume ls
4.创建Dockerfile
——安装openssh-server, openjdk-8-jdk, wget, vim, net-tools, iputils-ping, zookeeper, hadoop, hive, spark, Scala, SSH, Java等必要软件
——包括创建用户，设置机器名，ssh免密，环境变量，更换APT源为国内镜像。
4.1输入
mkdir -p ~/hadoop-cluster/{docker,scripts,data}
4.2输入
cd hadoop-cluster/docker4.3输入
vim Dockerfile.hadoop-ubuntu164.4输入以下内容，按下Esc输入:wq保存
FROM registry.cn-hangzhou.aliyuncs.com/acs/ubuntu:16.04
# 设置环境变量
ENV DEBIAN_FRONTEND=noninteractive
ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
ENV BIGDATA_HOME=/home/hadoop/bigdata
ENV HADOOP_HOME=/home/hadoop/bigdata/hadoop-3.3.4
ENV SPARK_HOME=/home/hadoop/bigdata/spark-3.3.4-bin-hadoop3
ENV HIVE_HOME=/home/hadoop/bigdata/apache-hive-3.1.3-bin
ENV ZOOKEEPER_HOME=/home/hadoop/bigdata/apache-zookeeper-3.7.2-bin
ENV SCALA_HOME=/home/hadoop/bigdata/scala-2.12.18
ENV MYSQL_HOME=/var/lib/mysql
ENV DATA_HOME=/home/hadoop/data
ENV PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin:$SCALA_HOME/bin
# 更换APT源为国内镜像（清华大学）
RUN cp /etc/apt/sources.list /etc/apt/sources.list.bak && \
echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial main restricted universe multiverse" > /etc/apt/sources.list && \
echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security main restricted universe multiverse" >> /etc/apt/sources.list
# 安装必要软件
RUN apt-get update && apt-get install -y \
openssh-server \
openjdk-8-jdk \
wget \
vim \
net-tools \
iputils-ping \
sudo \
passwd \
&& rm -rf /var/lib/apt/lists/*
# 配置SSH服务
RUN mkdir /var/run/sshd && \
ssh-keygen -A && \
sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config && \
sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/' /etc/ssh/sshd_config && \
sed -i 's/#AuthorizedKeysFile/AuthorizedKeysFile/' /etc/ssh/sshd_config
# 创建hadoop用户
RUN useradd -m -s /bin/bash hadoop && \
echo "hadoop:hadoop" | chpasswd && \
usermod -aG sudo hadoop
# 在root下创建目录并设置权限（hadoop用户没有chown权限）
RUN mkdir -p /home/hadoop/bigdata && \
mkdir -p /home/hadoop/data && \
chown -R hadoop:hadoop /home/hadoop
# 切换到hadoop用户
USER hadoop
WORKDIR /home/hadoop
# 检查目录
RUN ls -la /home/hadoop/ && \
ls -la /home/hadoop/bigdata
# 创建SSH密钥
RUN mkdir -p /home/hadoop/.ssh && \
ssh-keygen -t rsa -P "" -f /home/hadoop/.ssh/id_rsa && \
cat /home/hadoop/.ssh/id_rsa.pub > /home/hadoop/.ssh/authorized_keys && \
chmod 700 /home/hadoop/.ssh && \
chmod 600 /home/hadoop/.ssh/authorized_keys
# 安装Hadoop
RUN wget https://repo.huaweicloud.com/apache/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz && \
tar -vxzf hadoop-3.3.4.tar.gz -C /home/hadoop/bigdata && \
rm hadoop-3.3.4.tar.gz
# 安装Spark
RUN wget https://mirrors.huaweicloud.com/apache/spark/spark-3.3.4/spark-3.3.4-bin-hadoop3.tgz && \
tar -vxzf spark-3.3.4-bin-hadoop3.tgz -C /home/hadoop/bigdata && \
rm spark-3.3.4-bin-hadoop3.tgz
# 安装Hive
RUN wget https://mirrors.huaweicloud.com/apache/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz && \
tar -vxzf apache-hive-3.1.3-bin.tar.gz -C /home/hadoop/bigdata && \
rm apache-hive-3.1.3-bin.tar.gz
# 安装ZooKeeper
RUN wget https://mirrors.aliyun.com/apache/zookeeper/zookeeper-3.7.2/apache-zookeeper-3.7.2-bin.tar.gz && \
tar -vxzf apache-zookeeper-3.7.2-bin.tar.gz -C /home/hadoop/bigdata && \
rm apache-zookeeper-3.7.2-bin.tar.gz
# 安装Scala
RUN wget https://downloads.lightbend.com/scala/2.12.18/scala-2.12.18.tgz && \
tar -vxzf scala-2.12.18.tgz -C /home/hadoop/bigdata && \
rm scala-2.12.18.tgz
# 切换到root配置后续设置
USER root
# 创建数据目录结构
RUN mkdir -p ${DATA_HOME}/{hadoop/{tmp,hdfs/{name,data}},logs/{hadoop,spark},zookeeper,mysql} && \
chown -R hadoop:hadoop ${DATA_HOME}
# 配置环境变量到bashrc
RUN echo "export JAVA_HOME=${JAVA_HOME}" >> /etc/profile && \
echo "export HADOOP_HOME=${HADOOP_HOME}" >> /etc/profile && \
echo "export PATH=\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin" >> /etc/profile && \
echo "export SPARK_HOME=${SPARK_HOME}" >> /etc/profile && \
echo "export PATH=\$PATH:\$SPARK_HOME/bin" >> /etc/profile && \
echo "export HIVE_HOME=${HIVE_HOME}" >> /etc/profile && \
echo "export PATH=\$PATH:\$HIVE_HOME/bin" >> /etc/profile && \
echo "export SCALA_HOME=${SCALA_HOME}" >> /etc/profile && \
echo "export PATH=\$PATH:\$SCALA_HOME/bin" >> /etc/profile && \
echo "export ZOOKEEPER_HOME=${ZOOKEEPER_HOME}" >> /etc/profile && \
echo "export PATH=\$PATH:\$ZOOKEEPER_HOME/bin" >> /etc/profile && \
echo "export BIGDATA_HOME=${BIGDATA_HOME}" >> /etc/profile && \
echo "export DATA_HOME=${DATA_HOME}" >> /etc/profile
# 复制环境变量到hadoop用户的bashrc
RUN echo "source /etc/profile" >> /home/hadoop/.bashrc
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD ssh -o ConnectTimeout=5 hadoop@localhost exit 0 || exit 1
# 暴露端口
EXPOSE 22 8020 9870 8088 7077 8080 10000 3306 2181 2888 3888 9000 8030 8031 8032 8033 8040 8042
# 启动SSH服务
CMD ["/usr/sbin/sshd", "-D"]
4.5输入
podman build -t hadoop-cluster:ubuntu16 -f Dockerfile.hadoop-ubuntu16 .注意：最后面有一个"."
4.6输入
podman images
5.启动容器集群
5.1启动master，输入
podman run -d --name master \
--network hadoop-cluster \
--ip 10.99.0.10 \
--hostname master \
-p 20010:22 \
-p 9870:9870 -p 8088:8088 \
-p 7077:7077 -p 8080:8080 \
-p 10000:10000 -p 3306:3306 \
-p 19888:19888 -p 18080:18080 \
-v hadoop-all-data:/home/hadoop/data \
-v mysql-data:/var/lib/mysql \
hadoop-cluster:ubuntu16
5.2启动slave节点
5.2.1输入
podman run -d --name node1 \
--network hadoop-cluster \
--ip 10.99.0.11 \
--hostname node1 \
-p 20011:22 \
-v hadoop-all-data-1:/home/hadoop/data \
hadoop-cluster:ubuntu16
5.2.2输入
podman run -d --name node2 \
--network hadoop-cluster \
--ip 10.99.0.12 \
--hostname node2 \
-p 20012:22 \
-v hadoop-all-data-2:/home/hadoop/data \
hadoop-cluster:ubuntu16
5.2.3输入
podman run -d --name node3 \
--network hadoop-cluster \
--ip 10.99.0.13 \
--hostname node3 \
-p 20013:22 \
-v hadoop-all-data-3:/home/hadoop/data \
hadoop-cluster:ubuntu16
5.3验证容器状态，输入
podman ps -a
6.Master节点基础配置
6.1进入master，输入
podman exec -it master bash6.2进入hadoop账户输入
su - hadoop
6.3输入
sudo vim /etc/hosts最后一行修改成
10.99.0.10 master
10.99.0.11 node1
10.99.0.12 node2
10.99.0.13 node3
6.4分发hosts到所有Slave节点
6.4.1输入
ssh-copy-id hadoop@node16.4.2输入
ssh-copy-id hadoop@node26.4.3输入
ssh-copy-id hadoop@node3
6.4.4输入
cp /etc/hosts /tmp/hosts.new6.4.5免密
6.4.5.1输入
echo "hadoop ALL=(root) NOPASSWD:ALL" > /tmp/hadoop-nopasswd6.4.5.2输入
for n in node1 node2 node3; do
scp /tmp/hosts.new hadoop@$n:/tmp/hosts.new
ssh -t hadoop@$n "
sudo cp /tmp/hosts.new /etc/hosts &&
sudo rm -f /etc/sudoers.d/hadoop-nopasswd &&
sudo install -m 440 -o root -g root /tmp/hadoop-nopasswd /etc/sudoers.d/hadoop-nopasswd"
done
6.4.6验证
分别输入ssh node1, ssh node2, ssh node3
退出输入exit
7.Hadoop集群配置
7.1输入
cd /home/hadoop/bigdata/hadoop-3.3.4/etc/hadoop7.2输入
vim core-site.xml7.3在<configuration></configuration>之间输入以下内容，按下Esc输入:wq保存
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data/hadoop/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>node1:2181,node2:2181,node3:2181</value>
</property>
7.4输入
vim hdfs-site.xml7.5在<configuration></configuration>之间输入以下内容，按下Esc输入:wq保存
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>master:9870</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:9868</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
7.6输入
vim yarn-site.xml7.7在<configuration></configuration>之间输入以下内容，按下Esc输入:wq保存
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>
7.8输入
vim mapred-site.xml7.9在<configuration></configuration>之间输入以下内容，按下Esc输入:wq保存
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/hadoop/bigdata/hadoop-3.3.4</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/hadoop/bigdata/hadoop-3.3.4</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/hadoop/bigdata/hadoop-3.3.4</value>
</property>7.10输入
vim workers7.11删除localhost输入以下内容，按下Esc输入:wq保存
node1
node2
node37.12输入
vim ~/.bashrc7.13在开头输入以下内容，按下Esc输入:wq保存
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export BIGDATA_HOME=/home/hadoop/bigdata
export HADOOP_HOME=$BIGDATA_HOME/hadoop-3.3.4
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=$BIGDATA_HOME/spark-3.3.4-bin-hadoop3
export HIVE_HOME=$BIGDATA_HOME/apache-hive-3.1.3-bin
export ZOOKEEPER_HOME=$BIGDATA_HOME/apache-zookeeper-3.7.2-bin
export SCALA_HOME=$BIGDATA_HOME/scala-2.12.18
export DATA_HOME=/home/hadoop/data
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin:$SCALA_HOME/bin
7.14输入
source ~/.bashrc8.分发Hadoop配置
8.1输入
for node in node1 node2 node3; do
echo "Deploying to $node..."
scp -r /home/hadoop/bigdata/hadoop-3.3.4 $node:/home/hadoop/bigdata/
scp ~/.bashrc $node:/home/hadoop/
ssh $node "source ~/.bashrc"
done
8.2格式化，在master输入
hdfs namenode -format
9.ZooKeeper集群配置
9.1输入
for node in master node1 node2 node3; do
ssh $node "mkdir -p /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/{data,logs}"
done9.2创建myid文件
9.2.1输入
ssh node1 "echo '1' > /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/data/myid"9.2.2输入
ssh node2 "echo '2' > /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/data/myid"9.2.3输入
ssh node3 "echo '3' > /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/data/myid"9.2.4输入
echo '4' > /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/data/myid
9.3修改zoo.cfg
输入
cat > /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/zoo.cfg << 'EOF'
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/hadoop/data/apache-zookeeper-3.7.2-bin/data
dataLogDir=/home/hadoop/data/apache-zookeeper-3.7.2-bin/logs
clientPort=2181
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
server.4=master:2888:3888
EOF
9.4分发
9.4.1输入
scp /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/zoo.cfg node1:/home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/9.4.2输入
scp /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/zoo.cfg node2:/home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/9.4.3输入
scp /home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/zoo.cfg node3:/home/hadoop/bigdata/apache-zookeeper-3.7.2-bin/conf/
9.5启动ZooKeeper集群
9.5.1输入
zkServer.sh start9.5.2输入
for node in master node1 node2 node3; do
echo "starting $node ZooKeeper..."
ssh $node "zkServer.sh start"
done
9.6查看状态，输入
for node in master node1 node2 node3; do
echo "$node ZooKeeper status"
ssh $node "zkServer.sh status"
done
10.启动Hadoop集群
10.1启动HDFS，输入
start-dfs.sh
10.2启动YARN，输入
start-yarn.sh
10.3启动JobHistory，输入
mapred --daemon start historyserver10. 4验证集群状态
10.4.1输入
hdfs dfsadmin -report
10.4.2输入
yarn node -list
11.Hive安装配置
11.1 安装与配置MySQL
11.1.1输入
sudo apt update11.1.2输入
sudo apt install mysql-server11.1.3安装时候，设置root的密码root
11.1.4启动MySQL，输入
sudo service mysql start
11.1.5输入
sudo systemctl enable mysql
11.1.6输入
mysql -u root -p再输入密码root
11.1.7输入
CREATE DATABASE hivemetastore DEFAULT CHARACTER SET utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive2024';
GRANT ALL PRIVILEGES ON hivemetastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;
EXIT;
11.1.8绑定IP
11.1.8.1输入
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf11.1.8.2找到bind-address修改成bind-address = 0.0.0.0，按下Esc输入:wq保存
11.1.8.3输入
sudo -i11.1.8.4输入
service mysql restart11.1.8.5输入
su – hadoop
11.2配置Hive
11.2.1输入
cd /home/hadoop/bigdata/apache-hive-3.1.3-bin11.2.2输入
cd lib11.2.3输入
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar
11.2.4输入
cd ..11.2.5输入
cd conf11.2.6输入
vim hive-site.xml11.2.7输入以下内容，按下Esc输入:wq保存
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://master:3306/hivemetastore?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive2024</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://master:8020/user/hive/warehouse</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>master</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<property>
<name>hive.execution.engine</name>
<value>mr</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://master:9083</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/home/hadoop/data/hive/querylog</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/home/hadoop/data/hive/scratchdir</value>
</property>
<property>
<name>hive.exec.mode.local.auto</name>
<value>true</value>
</property>
<property>
<name>hive.log.dir</name>
<value>/home/hadoop/data/hive/logs</value>
</property>
</configuration>
11.2.8输入
schematool -initSchema -dbType mysql
12.Spark集群配置
12.1配置Spark
12.1.1输入
cd /home/hadoop/bigdata/spark-3.3.4-bin-hadoop3/conf12.1.2输入
vim spark-env.sh12.1.3输入以下内容，按下Esc输入:wq保存
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/home/hadoop/bigdata/hadoop-3.3.4
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_MASTER_HOST=master
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_MEMORY=2g
export SPARK_WORKER_CORES=2
export SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://master:8020/spark-logs"
export SPARK_DIST_CLASSPATH=$(hadoop classpath)12.1.4输入
vim spark-defaults.conf12.1.5输入以下内容，按下Esc输入:wq保存
spark.master spark://master:7077
spark.eventLog.enabled true
spark.eventLog.dir hdfs://master:8020/spark-logs
spark.history.fs.logDirectory hdfs://master:8020/spark-logs
spark.sql.warehouse.dir hdfs://master:8020/user/hive/warehouse
spark.yarn.jars hdfs://master:8020/spark-jars/*.jar
spark.hadoop.fs.defaultFS hdfs://master:802012.1.6输入
vim workers12.1.7输入以下内容，按下Esc输入:wq保存
node1
node2
node312.2创建HDFS目录
12.2.1输入
hdfs dfs -mkdir -p /spark-logs12.2.2输入
hdfs dfs -mkdir -p /spark-jars12.2.3输入
hdfs dfs -put /home/hadoop/bigdata/spark-3.3.4-bin-hadoop3/jars/*.jar /spark-jars/
12.2.4输入
hdfs dfs -chmod -R 777 /spark-logs12.2.5输入
hdfs dfs -chmod -R 755 /spark-jars12.2.6输入
hdfs dfs -ls /
12.3分发Spark配置到node
for node in node1 node2 node3; do
echo "Deploy Spark to $node..."
scp -r /home/hadoop/bigdata/spark-3.3.4-bin-hadoop3 hadoop@$node:/home/hadoop/bigdata/
done
12.4启动Spark集群
12.4.1输入
$SPARK_HOME/sbin/start-all.sh
12.4.2输入
$SPARK_HOME/sbin/start-history-server.sh
12.4.3输入
jps
14.虚拟机快照
--容器关闭之后或者虚拟机关机之后Podman集群会丢失所有数据，可以使用卷持久化数据，或者使用快照回滚
14.1快照
14.1.1右键虚拟机，找到快照，快照管理器（14.1.1-A），或者点击拍摄快照
14.1.1-A快照管理器，点击拍摄快照，修改名称
14.1.1-B
14.1.1-C点击转到，就可以回到所选的快照状态，恢复集群数据
14.2暂停Podman
--暂停之后重启电脑仍然会丢失集群数据，如果需要关闭虚拟机则需要使用快照
14.2.1输入exit退出
14.2.2输入
podman ps
14.2.3输入
podman pause $(podman ps -q)
输入
podman ps -a查看状态变成Paused
14.2.4如果想再次启动可以输入
——宿主机没有关机或者重启可以集训运行，若已经关机或重启数据会丢失状态会变成Created
podman unpause master
podman unpause node1
podman unpause node2
podman unpause node3注意：如果暂停过Podman，需要重新启动集群，才能进行集群验证测试
15.集群验证测试
15.1输入
hdfs dfs -mkdir -p /user/hadoop15.2输入
hdfs dfs -mkdir test15.3输入
hdfs dfs -mkdir test/input15.4输入
hdfs dfs -ls
15.5输入
echo "hello hadoop world" > test.txt15.6输入
echo "spark is awesome" >> test.txtcat test.txt
15.7输入
hdfs dfs -put test.txt test/input/15.8输入
hdfs dfs -cat test/input/test.txt
15.9输入
hadoop jar /home/hadoop/bigdata/hadoop-3.3.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.4.jar wordcount test/input test/output
15.10查看结果
15.10.1输入
hdfs dfs -cat test/output/part-r-00000
15.11测试Spark
--运行估算圆周率π
15.11.1输入
spark-submit --master spark://master:7077 --class org.apache.spark.examples.SparkPi /home/hadoop/bigdata/spark-3.3.4-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.4.jar 100
15.11.2查看结果，在输出的日志找到
15.12测试Hive
15.12.1输入
nohup hive --service metastore > hive-metastore.log 2>&1 &
15.12.2输入
jps
15.12.3输入
hive
15.12.4输入
CREATE TABLE test_table (id INT, name STRING);
15.12.5输入
INSERT INTO test_table VALUES (1, 'hadoop'), (2, 'spark');
15.12.6输入
SELECT * FROM test_table;
输入
quit;退出
三、遇到的问题与解决
1.无法ssh
解决：输入
ssh-keygen -R <自己配置的IP>
2.无法连接
解决：修改 Dockerfile 的第一行
FROM registry.cn-hangzhou.aliyuncs.com/acs/ubuntu:16.04
3.镜像失效
解决：更改镜像，执行
podman rmi $(podman images -f dangling=true -q) 2>/dev/null || true，再重新构建
4.3306端口被占用
解决：关闭宿主机MySQL
5.
解决：授权
6.Hive初始化失败，MySQL拒绝访问
解决：
vim /etc/mysql/mysql.conf.d/mysqld.cnf找到bind-address修改成bind-address = 0.0.0.0，按下Esc输入:wq保存
7.Hive创建表失败
解决：
nohup hive --service metastore > hive-metastore.log 2>&1 &9.MySQL使用systemctl指令无法启动
报错Failed to connect to bus: No such file or directory
解决：使用sudo service mysql start指令启动
10.Spark重启失败，或者使用集群计算失败
解决：在各节点使用以下指令，清除缓存之后再尝试启动Spark
cd /home/hadoop/bigdata/spark-3.3.4-bin-hadoop3rm -rf /tmp/spark-*rm -rf /tmp/blockmgr-*rm -rf /tmp/spark.log更多推荐
13
0- 0
已为社区贡献4条内容
所有评论(0)