k8s部署hadoop
配置和模板参考helm仓库:https://artifacthub.io/packages/helm/apache-hadoop-helm/hadoop。
·
(作者:陈玓玏)
配置和模板参考helm仓库:https://artifacthub.io/packages/helm/apache-hadoop-helm/hadoop
先通过以下命令生成yaml文件:
helm template hadoop pfisterer-hadoop/hadoop > hadoop.yaml
用kubectl命令,通过这个yaml文件直接部署hadoop,配置想改的可以自己改,默认的配置仓库中有写:
比如我想要3个datanode而不是1个,就把dn-statefulset的replicas改为3,保存后直接部署,部署命令:kubectl apply -f hadoop.yaml
hadoop.yaml中,有四个需要修改的地方,不然之后用hive和spark会出问题。
- 修改hadoop-hadoop-hdfs-dn这个StatefulSet下的空目录
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
- name: dfs
emptyDir: {}
改为:
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
- name: dfs
hostPath:
path: /data/k8s/hdfs/datanode
type: Directory
不执行这一步操作,可能出现hdfs没有持久化存储的问题,每次hdfs的pod一重启,数据就会丢失,所以需要修改这个挂载。
除了datanode,还有一个namenode的emptyDir要改。
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
- name: dfs
hostPath:
path: /data/k8s/hdfs/namenode
type: Directory
- yaml中还需要把slaves改为workers,要把所有的slaves都替换成workers。
- 修改yarn-site.xml,配置以下内容:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8030</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
- 去掉resource的limits,资源限制得太小了,要么就改大点,不然hive和spark跑数据跑不起来。
k8sdashboard中可以看到3个dn节点:
部署之后,进入namenode的pod,就可以执行hdfs命令了,如hdfs dfs -ls,hdfs dfs -touch等等。
修改后的完整yaml参考:
---
# Source: hadoop/templates/hdfs-dn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hadoop-hadoop-hdfs-dn
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
minAvailable: 1
---
# Source: hadoop/templates/hdfs-nn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hadoop-hadoop-hdfs-nn
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
minAvailable: 1
---
# Source: hadoop/templates/yarn-nm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hadoop-hadoop-yarn-nm
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
minAvailable: 1
---
# Source: hadoop/templates/yarn-rm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: hadoop-hadoop-yarn-rm
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
minAvailable: 1
---
# Source: hadoop/templates/hadoop-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: hadoop-hadoop
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
data:
bootstrap.sh: |
#!/bin/bash -x
echo Starting
: ${HADOOP_HOME:=/opt/hadoop}
echo Using ${HADOOP_HOME} as HADOOP_HOME
. $HADOOP_HOME/etc/hadoop/hadoop-env.sh
# ------------------------------------------------------
# Directory to find config artifacts
# ------------------------------------------------------
CONFIG_DIR="/tmp/hadoop-config"
# ------------------------------------------------------
# Copy config files from volume mount
# ------------------------------------------------------
for f in workers core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml; do
if [[ -e ${CONFIG_DIR}/$f ]]; then
cp ${CONFIG_DIR}/$f $HADOOP_HOME/etc/hadoop/$f
else
echo "ERROR: Could not find $f in $CONFIG_DIR"
exit 1
fi
done
# ------------------------------------------------------
# installing libraries if any
# (resource urls added comma separated to the ACP system variable)
# ------------------------------------------------------
cd $HADOOP_HOME/share/hadoop/common ; for cp in ${ACP//,/ }; do echo == $cp; curl -LO $cp ; done; cd -
# ------------------------------------------------------
# Start NAMENODE
# ------------------------------------------------------
if [[ "${HOSTNAME}" =~ "hdfs-nn" ]]; then
# sed command changing REPLACEME in $HADOOP_HOME/etc/hadoop/hdfs-site.xml to actual port numbers
sed -i "s/EXTERNAL_HTTP_PORT_REPLACEME/9864/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
sed -i "s/EXTERNAL_DATA_PORT_REPLACEME/9866/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
mkdir -p /root/hdfs/namenode
if [ ! -f /root/hdfs/namenode/formated ]; then
# Only format if necessary
$HADOOP_HOME/bin/hdfs namenode -format -force -nonInteractive && echo 1 > /root/hdfs/namenode/formated
fi
$HADOOP_HOME/bin/hdfs --loglevel INFO --daemon start namenode
fi
# ------------------------------------------------------
# Start DATA NODE
# ------------------------------------------------------
if [[ "${HOSTNAME}" =~ "hdfs-dn" ]]; then
# Split hostname at "-" into an array
# Example hostname: hadoop-hadoop-hdfs-dn-0
HOSTNAME_ARR=(${HOSTNAME//-/ })
# Add instance number to start of external port ranges
EXTERNAL_HTTP_PORT=$((51000 + ${HOSTNAME_ARR[4]}))
EXTERNAL_DATA_PORT=$((50500 + ${HOSTNAME_ARR[4]}))
# sed command changing REPLACEME in $HADOOP_HOME/etc/hadoop/hdfs-site.xml to actual port numbers
sed -i "s/EXTERNAL_HTTP_PORT_REPLACEME/${EXTERNAL_HTTP_PORT}/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
sed -i "s/EXTERNAL_DATA_PORT_REPLACEME/${EXTERNAL_DATA_PORT}/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
mkdir -p /root/hdfs/datanode
# Wait (with timeout) for namenode
TMP_URL="http://hadoop-hadoop-hdfs-nn:9870"
if timeout 5m bash -c "until curl -sf $TMP_URL; do echo Waiting for $TMP_URL; sleep 5; done"; then
$HADOOP_HOME/bin/hdfs --loglevel INFO --daemon start datanode
else
echo "$0: Timeout waiting for $TMP_URL, exiting."
exit 1
fi
fi
# ------------------------------------------------------
# Start RESOURCE MANAGER and PROXY SERVER as daemons
# ------------------------------------------------------
if [[ "${HOSTNAME}" =~ "yarn-rm" ]]; then
$HADOOP_HOME/bin/yarn --loglevel INFO --daemon start resourcemanager
$HADOOP_HOME/bin/yarn --loglevel INFO --daemon start proxyserver
fi
# ------------------------------------------------------
# Start NODE MANAGER
# ------------------------------------------------------
if [[ "${HOSTNAME}" =~ "yarn-nm" ]]; then
sed -i '/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/yarn-site.xml
cat >> $HADOOP_HOME/etc/hadoop/yarn-site.xml <<- EOM
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${MY_MEM_LIMIT:-2048}</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${MY_CPU_LIMIT:-2}</value>
</property>
EOM
echo '</configuration>' >> $HADOOP_HOME/etc/hadoop/yarn-site.xml
# Wait with timeout for resourcemanager
TMP_URL="http://hadoop-hadoop-yarn-rm:8088/ws/v1/cluster/info"
if timeout 5m bash -c "until curl -sf $TMP_URL; do echo Waiting for $TMP_URL; sleep 5; done"; then
$HADOOP_HOME/bin/yarn nodemanager --loglevel INFO
else
echo "$0: Timeout waiting for $TMP_URL, exiting."
exit 1
fi
fi
# ------------------------------------------------------
# Tail logfiles for daemonized workloads (parameter -d)
# ------------------------------------------------------
if [[ $1 == "-d" ]]; then
until find ${HADOOP_HOME}/logs -mmin -1 | egrep -q '.*'; echo "`date`: Waiting for logs..." ; do sleep 2 ; done
tail -F ${HADOOP_HOME}/logs/* &
while true; do sleep 1000; done
fi
# ------------------------------------------------------
# Start bash if requested (parameter -bash)
# ------------------------------------------------------
if [[ $1 == "-bash" ]]; then
/bin/bash
fi
core-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-hadoop-hdfs-nn:9000/</value>
<description>NameNode URI</description>
</property>
</configuration>
hdfs-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property><property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.hostname</name>
<value>example.com</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:EXTERNAL_HTTP_PORT_REPLACEME</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:EXTERNAL_DATA_PORT_REPLACEME</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///root/hdfs/datanode</value>
<description>DataNode directory</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///root/hdfs/namenode</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<!-- Bind to all interfaces -->
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>dfs.namenode.servicerpc-bind-host</name>
<value>0.0.0.0</value>
</property>
<!-- /Bind to all interfaces -->
</configuration>
mapred-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default.svc.cluster.local:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default.svc.cluster.local:19888</value>
</property>
</configuration>
workers: |
localhost
yarn-site.xml: |
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8030</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
<!-- Bind to all interfaces -->
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.timeline-service.bind-host</name>
<value>0.0.0.0</value>
</property>
<!-- /Bind to all interfaces -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<description>List of directories to store localized files in.</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/var/log/hadoop-yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
/opt/hadoop/etc/hadoop,
/opt/hadoop/share/hadoop/common/*,
/opt/hadoop/share/hadoop/common/lib/*,
/opt/hadoop/share/hadoop/hdfs/*,
/opt/hadoop/share/hadoop/hdfs/lib/*,
/opt/hadoop/share/hadoop/mapreduce/*,
/opt/hadoop/share/hadoop/mapreduce/lib/*,
/opt/hadoop/share/hadoop/yarn/*,
/opt/hadoop/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>
---
# Source: hadoop/templates/hdfs-dn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: hadoop-hadoop-hdfs-dn
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
spec:
ports:
- name: dfs
port: 9000
protocol: TCP
# - name: webhdfs
# port: 9864
clusterIP: None
selector:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
---
# Source: hadoop/templates/hdfs-nn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: hadoop-hadoop-hdfs-nn
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
spec:
ports:
- name: dfs
port: 9000
protocol: TCP
- name: webhdfs
port: 9870
clusterIP: None
selector:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
---
# Source: hadoop/templates/yarn-nm-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: hadoop-hadoop-yarn-nm
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
spec:
ports:
- port: 8088
name: web
- port: 8082
name: web2
- port: 8042
name: api
clusterIP: None
selector:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
---
# Source: hadoop/templates/yarn-rm-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
name: hadoop-hadoop-yarn-rm
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
spec:
ports:
- port: 8088
name: web
clusterIP: None
selector:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/yarn-ui-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
name: hadoop-hadoop-yarn-ui
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-ui
spec:
ports:
- port: 8088
name: web
selector:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/hdfs-dn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hadoop-hadoop-hdfs-dn
annotations:
checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
serviceName: hadoop-hadoop-hdfs-dn
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-dn
terminationGracePeriodSeconds: 0
containers:
- name: hdfs-dn
image: "farberg/apache-hadoop:3.3.2"
imagePullPolicy: "IfNotPresent"
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
- "-d"
resources:
# readinessProbe:
# httpGet:
# path: /
# port: 9864
# initialDelaySeconds: 60
# timeoutSeconds: 2
# livenessProbe:
# httpGet:
# path: /
# port: 9864
# initialDelaySeconds: 60
# timeoutSeconds: 2
volumeMounts:
- name: hadoop-config
mountPath: /tmp/hadoop-config
- name: dfs
mountPath: /root/hdfs/datanode
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
- name: dfs
hostPath:
path: /data/k8s/hdfs/datanode
type: Directory
---
# Source: hadoop/templates/hdfs-nn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hadoop-hadoop-hdfs-nn
annotations:
checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
serviceName: hadoop-hadoop-hdfs-nn
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: hdfs-nn
terminationGracePeriodSeconds: 0
containers:
- name: hdfs-nn
image: "farberg/apache-hadoop:3.3.2"
imagePullPolicy: "IfNotPresent"
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
- "-d"
resources:
readinessProbe:
httpGet:
path: /
port: 9870
initialDelaySeconds: 60
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /
port: 9870
initialDelaySeconds: 60
timeoutSeconds: 2
volumeMounts:
- name: hadoop-config
mountPath: /tmp/hadoop-config
- name: dfs
mountPath: /root/hdfs/namenode
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
- name: dfs
hostPath:
path: /data/k8s/hdfs/namenode
type: Directory
---
# Source: hadoop/templates/yarn-nm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hadoop-hadoop-yarn-nm
annotations:
checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
serviceName: hadoop-hadoop-yarn-nm
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-nm
terminationGracePeriodSeconds: 0
containers:
- name: yarn-nm
image: "farberg/apache-hadoop:3.3.2"
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 8088
name: web
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
- "-d"
resources:
readinessProbe:
httpGet:
path: /node
port: 8042
initialDelaySeconds: 10
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /node
port: 8042
initialDelaySeconds: 10
timeoutSeconds: 2
env:
- name: MY_CPU_LIMIT
valueFrom:
resourceFieldRef:
containerName: yarn-nm
resource: limits.cpu
divisor: 1
- name: MY_MEM_LIMIT
valueFrom:
resourceFieldRef:
containerName: yarn-nm
resource: limits.memory
divisor: 1M
volumeMounts:
- name: hadoop-config
mountPath: /tmp/hadoop-config
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
---
# Source: hadoop/templates/yarn-rm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hadoop-hadoop-yarn-rm
annotations:
checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
labels:
app.kubernetes.io/name: hadoop
helm.sh/chart: hadoop-1.2.0
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
spec:
selector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
serviceName: hadoop-hadoop-yarn-rm
replicas: 1
template:
metadata:
labels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 5
podAffinityTerm:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app.kubernetes.io/name: hadoop
app.kubernetes.io/instance: hadoop
app.kubernetes.io/component: yarn-rm
terminationGracePeriodSeconds: 0
containers:
- name: yarn-rm
image: "farberg/apache-hadoop:3.3.2"
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 8088
name: web
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
- "-d"
resources:
readinessProbe:
httpGet:
path: /ws/v1/cluster/info
port: 8088
initialDelaySeconds: 5
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /ws/v1/cluster/info
port: 8088
initialDelaySeconds: 10
timeoutSeconds: 2
volumeMounts:
- name: hadoop-config
mountPath: /tmp/hadoop-config
volumes:
- name: hadoop-config
configMap:
name: hadoop-hadoop
更多推荐
已为社区贡献11条内容
所有评论(0)