（作者：陈玓玏）



配置和模板参考helm仓库：https://artifacthub.io/packages/helm/apache-hadoop-helm/hadoop

先通过以下命令生成yaml文件：

helm template hadoop pfisterer-hadoop/hadoop  > hadoop.yaml

在这里插入图片描述
用kubectl命令，通过这个yaml文件直接部署hadoop，配置想改的可以自己改，默认的配置仓库中有写：
在这里插入图片描述
比如我想要3个datanode而不是1个，就把dn-statefulset的replicas改为3，保存后直接部署，部署命令：kubectl apply -f hadoop.yaml
在这里插入图片描述
hadoop.yaml中，有四个需要修改的地方，不然之后用hive和spark会出问题。

  1. 修改hadoop-hadoop-hdfs-dn这个StatefulSet下的空目录
   volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
      - name: dfs
        emptyDir: {}

改为：

   volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
      - name: dfs
        hostPath:
          path: /data/k8s/hdfs/datanode
          type: Directory

不执行这一步操作，可能出现hdfs没有持久化存储的问题，每次hdfs的pod一重启，数据就会丢失，所以需要修改这个挂载。
除了datanode，还有一个namenode的emptyDir要改。

   volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
      - name: dfs
        hostPath:
          path: /data/k8s/hdfs/namenode
          type: Directory
  1. yaml中还需要把slaves改为workers，要把所有的slaves都替换成workers。
  2. 修改yarn-site.xml，配置以下内容：
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8032</value>
    </property>
        <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8030</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>
  1. 去掉resource的limits，资源限制得太小了，要么就改大点，不然hive和spark跑数据跑不起来。

k8sdashboard中可以看到3个dn节点：
在这里插入图片描述
部署之后，进入namenode的pod，就可以执行hdfs命令了，如hdfs dfs -ls，hdfs dfs -touch等等。
在这里插入图片描述

修改后的完整yaml参考：

---
# Source: hadoop/templates/hdfs-dn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hdfs-dn
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-dn
  minAvailable: 1
---
# Source: hadoop/templates/hdfs-nn-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-hdfs-nn
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-nn
  minAvailable: 1
---
# Source: hadoop/templates/yarn-nm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-yarn-nm
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-nm
  minAvailable: 1
---
# Source: hadoop/templates/yarn-rm-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hadoop-hadoop-yarn-rm
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-rm
  minAvailable: 1
---
# Source: hadoop/templates/hadoop-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: hadoop-hadoop
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
data:
  bootstrap.sh: |
    #!/bin/bash -x

    echo Starting

    : ${HADOOP_HOME:=/opt/hadoop}

    echo Using ${HADOOP_HOME} as HADOOP_HOME

    . $HADOOP_HOME/etc/hadoop/hadoop-env.sh

    # ------------------------------------------------------
    # Directory to find config artifacts
    # ------------------------------------------------------

    CONFIG_DIR="/tmp/hadoop-config"

    # ------------------------------------------------------
    # Copy config files from volume mount
    # ------------------------------------------------------

    for f in workers core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml; do
      if [[ -e ${CONFIG_DIR}/$f ]]; then
        cp ${CONFIG_DIR}/$f $HADOOP_HOME/etc/hadoop/$f
      else
        echo "ERROR: Could not find $f in $CONFIG_DIR"
        exit 1
      fi
    done

    # ------------------------------------------------------
    # installing libraries if any
    # (resource urls added comma separated to the ACP system variable)
    # ------------------------------------------------------
    cd $HADOOP_HOME/share/hadoop/common ; for cp in ${ACP//,/ }; do  echo == $cp; curl -LO $cp ; done; cd -

    # ------------------------------------------------------
    # Start NAMENODE
    # ------------------------------------------------------
    if [[ "${HOSTNAME}" =~ "hdfs-nn" ]]; then
      # sed command changing REPLACEME in $HADOOP_HOME/etc/hadoop/hdfs-site.xml to actual port numbers
      sed -i "s/EXTERNAL_HTTP_PORT_REPLACEME/9864/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
      sed -i "s/EXTERNAL_DATA_PORT_REPLACEME/9866/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml

      mkdir -p /root/hdfs/namenode
      if [ ! -f /root/hdfs/namenode/formated ]; then
        # Only format if necessary
        $HADOOP_HOME/bin/hdfs namenode -format -force -nonInteractive && echo 1 > /root/hdfs/namenode/formated
      fi
      $HADOOP_HOME/bin/hdfs --loglevel INFO --daemon start namenode
    fi

    # ------------------------------------------------------
    # Start DATA NODE
    # ------------------------------------------------------
    if [[ "${HOSTNAME}" =~ "hdfs-dn" ]]; then
      # Split hostname at "-" into an array
      # Example hostname: hadoop-hadoop-hdfs-dn-0
      HOSTNAME_ARR=(${HOSTNAME//-/ })
      # Add instance number to start of external port ranges
      EXTERNAL_HTTP_PORT=$((51000 + ${HOSTNAME_ARR[4]}))
      EXTERNAL_DATA_PORT=$((50500 + ${HOSTNAME_ARR[4]}))

      # sed command changing REPLACEME in $HADOOP_HOME/etc/hadoop/hdfs-site.xml to actual port numbers
      sed -i "s/EXTERNAL_HTTP_PORT_REPLACEME/${EXTERNAL_HTTP_PORT}/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml
      sed -i "s/EXTERNAL_DATA_PORT_REPLACEME/${EXTERNAL_DATA_PORT}/" $HADOOP_HOME/etc/hadoop/hdfs-site.xml

      mkdir -p /root/hdfs/datanode

      #  Wait (with timeout) for namenode
      TMP_URL="http://hadoop-hadoop-hdfs-nn:9870"
      if timeout 5m bash -c "until curl -sf $TMP_URL; do echo Waiting for $TMP_URL; sleep 5; done"; then
        $HADOOP_HOME/bin/hdfs --loglevel INFO --daemon start datanode
      else 
        echo "$0: Timeout waiting for $TMP_URL, exiting."
        exit 1
      fi

    fi

    # ------------------------------------------------------
    # Start RESOURCE MANAGER and PROXY SERVER as daemons
    # ------------------------------------------------------
    if [[ "${HOSTNAME}" =~ "yarn-rm" ]]; then
      $HADOOP_HOME/bin/yarn --loglevel INFO --daemon start resourcemanager 
      $HADOOP_HOME/bin/yarn --loglevel INFO --daemon start proxyserver
    fi

    # ------------------------------------------------------
    # Start NODE MANAGER
    # ------------------------------------------------------
    if [[ "${HOSTNAME}" =~ "yarn-nm" ]]; then
      sed -i '/<\/configuration>/d' $HADOOP_HOME/etc/hadoop/yarn-site.xml
      cat >> $HADOOP_HOME/etc/hadoop/yarn-site.xml <<- EOM
      <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>${MY_MEM_LIMIT:-2048}</value>
      </property>

      <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>${MY_CPU_LIMIT:-2}</value>
      </property>
    EOM

      echo '</configuration>' >> $HADOOP_HOME/etc/hadoop/yarn-site.xml

      # Wait with timeout for resourcemanager
      TMP_URL="http://hadoop-hadoop-yarn-rm:8088/ws/v1/cluster/info"
      if timeout 5m bash -c "until curl -sf $TMP_URL; do echo Waiting for $TMP_URL; sleep 5; done"; then
        $HADOOP_HOME/bin/yarn nodemanager --loglevel INFO
      else 
        echo "$0: Timeout waiting for $TMP_URL, exiting."
        exit 1
      fi

    fi

    # ------------------------------------------------------
    # Tail logfiles for daemonized workloads (parameter -d)
    # ------------------------------------------------------
    if [[ $1 == "-d" ]]; then
      until find ${HADOOP_HOME}/logs -mmin -1 | egrep -q '.*'; echo "`date`: Waiting for logs..." ; do sleep 2 ; done
      tail -F ${HADOOP_HOME}/logs/* &
      while true; do sleep 1000; done
    fi

    # ------------------------------------------------------
    # Start bash if requested (parameter -bash)
    # ------------------------------------------------------
    if [[ $1 == "-bash" ]]; then
      /bin/bash
    fi

  core-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
      <property>
            <name>fs.defaultFS</name>
            <value>hdfs://hadoop-hadoop-hdfs-nn:9000/</value>
            <description>NameNode URI</description>
        </property>
    </configuration>

  hdfs-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration><property>
          <name>dfs.webhdfs.enabled</name>
          <value>true</value>
      </property><property>
        <name>dfs.datanode.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.client.use.datanode.hostname</name>
        <value>false</value>
      </property>

      <property>
        <name>dfs.datanode.hostname</name>
        <value>example.com</value>
      </property>

      <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:EXTERNAL_HTTP_PORT_REPLACEME</value>
      </property>

      <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:EXTERNAL_DATA_PORT_REPLACEME</value>
      </property>

      <property>
        <name>dfs.replication</name>
          <value>3</value>
      </property>

      <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///root/hdfs/datanode</value>
        <description>DataNode directory</description>
      </property>

      <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///root/hdfs/namenode</value>
        <description>NameNode directory for namespace and transaction logs storage.</description>
      </property>

      <property>
        <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
        <value>false</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>dfs.namenode.rpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>dfs.namenode.servicerpc-bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

    </configuration>

  mapred-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>
      <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
      </property>
      <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default.svc.cluster.local:10020</value>
      </property>
      <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default.svc.cluster.local:19888</value>
      </property>
    </configuration>

  workers: |
    localhost

  yarn-site.xml: |
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <configuration>
      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>
      <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default</value>
      </property>
      <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8032</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-hadoop-yarn-rm-0.hadoop-hadoop-yarn-rm.default:8030</value>
      </property>
      <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
      </property>

      <!-- Bind to all interfaces -->
      <property>
        <name>yarn.resourcemanager.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>yarn.nodemanager.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <property>
        <name>yarn.timeline-service.bind-host</name>
        <value>0.0.0.0</value>
      </property>
      <!-- /Bind to all interfaces -->

      <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
      </property>

      <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
      </property>

      <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
      </property>

      <property>
        <description>List of directories to store localized files in.</description>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
      </property>

      <property>
        <description>Where to store container logs.</description>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/var/log/hadoop-yarn/containers</value>
      </property>

      <property>
        <description>Where to aggregate logs to.</description>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/var/log/hadoop-yarn/apps</value>
      </property>

      <property>
        <name>yarn.application.classpath</name>
        <value>
          /opt/hadoop/etc/hadoop,
          /opt/hadoop/share/hadoop/common/*,
          /opt/hadoop/share/hadoop/common/lib/*,
          /opt/hadoop/share/hadoop/hdfs/*,
          /opt/hadoop/share/hadoop/hdfs/lib/*,
          /opt/hadoop/share/hadoop/mapreduce/*,
          /opt/hadoop/share/hadoop/mapreduce/lib/*,
          /opt/hadoop/share/hadoop/yarn/*,
          /opt/hadoop/share/hadoop/yarn/lib/*
        </value>
      </property>
    </configuration>
---
# Source: hadoop/templates/hdfs-dn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-dn
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  ports:
  - name: dfs
    port: 9000
    protocol: TCP
  # - name: webhdfs
  #   port: 9864
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
---
# Source: hadoop/templates/hdfs-nn-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-hdfs-nn
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: dfs
    port: 9000
    protocol: TCP
  - name: webhdfs
    port: 9870
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
---
# Source: hadoop/templates/yarn-nm-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-nm
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  ports:
  - port: 8088
    name: web
  - port: 8082
    name: web2
  - port: 8042
    name: api
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
---
# Source: hadoop/templates/yarn-rm-svc.yaml
# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-rm
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  ports:
  - port: 8088
    name: web
  clusterIP: None
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/yarn-ui-svc.yaml
# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: hadoop-hadoop-yarn-ui
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-ui
spec:
  ports:
  - port: 8088
    name: web
  selector:
    app.kubernetes.io/name: hadoop
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
---
# Source: hadoop/templates/hdfs-dn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hdfs-dn
  annotations:
    checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-dn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-dn
  serviceName: hadoop-hadoop-hdfs-dn
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hdfs-dn
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hdfs-dn
      terminationGracePeriodSeconds: 0
      containers:
      - name: hdfs-dn
        image: "farberg/apache-hadoop:3.3.2"
        imagePullPolicy: "IfNotPresent"
        command:
           - "/bin/bash"
           - "/tmp/hadoop-config/bootstrap.sh"
           - "-d"
        resources:
        # readinessProbe:
        #   httpGet:
        #     path: /
        #     port: 9864
        #   initialDelaySeconds: 60
        #   timeoutSeconds: 2
        # livenessProbe:
        #   httpGet:
        #     path: /
        #     port: 9864
        #   initialDelaySeconds: 60
        #   timeoutSeconds: 2
        volumeMounts:
        - name: hadoop-config
          mountPath: /tmp/hadoop-config
        - name: dfs
          mountPath: /root/hdfs/datanode
      volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
      - name: dfs
        hostPath:
          path: /data/k8s/hdfs/datanode
          type: Directory
---
# Source: hadoop/templates/hdfs-nn-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-hdfs-nn
  annotations:
    checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: hdfs-nn
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: hdfs-nn
  serviceName: hadoop-hadoop-hdfs-nn
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: hdfs-nn
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: hdfs-nn
      terminationGracePeriodSeconds: 0
      containers:
      - name: hdfs-nn
        image: "farberg/apache-hadoop:3.3.2"
        imagePullPolicy: "IfNotPresent"
        command:
        - "/bin/bash"
        - "/tmp/hadoop-config/bootstrap.sh"
        - "-d"
        resources:
        readinessProbe:
          httpGet:
            path: /
            port: 9870
          initialDelaySeconds: 60
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /
            port: 9870
          initialDelaySeconds: 60
          timeoutSeconds: 2
        volumeMounts:
        - name: hadoop-config
          mountPath: /tmp/hadoop-config
        - name: dfs
          mountPath: /root/hdfs/namenode
      volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
      - name: dfs
        hostPath:
          path: /data/k8s/hdfs/namenode
          type: Directory
---
# Source: hadoop/templates/yarn-nm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-yarn-nm
  annotations:
    checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-nm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-nm
  serviceName: hadoop-hadoop-yarn-nm
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: yarn-nm
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: yarn-nm
      terminationGracePeriodSeconds: 0
      containers:
      - name: yarn-nm
        image: "farberg/apache-hadoop:3.3.2"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 8088
          name: web
        command:
           - "/bin/bash"
           - "/tmp/hadoop-config/bootstrap.sh"
           - "-d"
        resources:
        readinessProbe:
          httpGet:
            path: /node
            port: 8042
          initialDelaySeconds: 10
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /node
            port: 8042
          initialDelaySeconds: 10
          timeoutSeconds: 2
        env:
        - name: MY_CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: yarn-nm
              resource: limits.cpu
              divisor: 1
        - name: MY_MEM_LIMIT
          valueFrom:
            resourceFieldRef:
              containerName: yarn-nm
              resource: limits.memory
              divisor: 1M
        volumeMounts:
        - name: hadoop-config
          mountPath: /tmp/hadoop-config
      volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
---
# Source: hadoop/templates/yarn-rm-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hadoop-hadoop-yarn-rm
  annotations:
    checksum/config: 79e26cb9566573abbcc0729e486c72f706591c0b6d99e3f26aa3ba8e3ecf9bf1
  labels:
    app.kubernetes.io/name: hadoop
    helm.sh/chart: hadoop-1.2.0
    app.kubernetes.io/instance: hadoop
    app.kubernetes.io/component: yarn-rm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hadoop
      app.kubernetes.io/instance: hadoop
      app.kubernetes.io/component: yarn-rm
  serviceName: hadoop-hadoop-yarn-rm
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hadoop
        app.kubernetes.io/instance: hadoop
        app.kubernetes.io/component: yarn-rm
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 5
            podAffinityTerm:
              topologyKey: "kubernetes.io/hostname"
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: hadoop
                  app.kubernetes.io/instance: hadoop
                  app.kubernetes.io/component: yarn-rm
      terminationGracePeriodSeconds: 0
      containers:
      - name: yarn-rm
        image: "farberg/apache-hadoop:3.3.2"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 8088
          name: web
        command:
           - "/bin/bash"
           - "/tmp/hadoop-config/bootstrap.sh"
           - "-d"
        resources:
        readinessProbe:
          httpGet:
            path: /ws/v1/cluster/info
            port: 8088
          initialDelaySeconds: 5
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /ws/v1/cluster/info
            port: 8088
          initialDelaySeconds: 10
          timeoutSeconds: 2
        volumeMounts:
        - name: hadoop-config
          mountPath: /tmp/hadoop-config
      volumes:
      - name: hadoop-config
        configMap:
          name: hadoop-hadoop
# kubernetes # hadoop # 容器
Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes