YARN学习总结-第十九节-YARN-Service

YARN-ServiceYARN-Service提供一流的支持和API用户托管长期运行的服务。简而言之，它充当容器编排平台管理容器化服务。同时支持Docker容器和传统在YARN之上基于进程的容器。这个框架的功能包括执行配置解决方案和挂载，生命周期管理，例如停止、启动、删除服务，伸缩服务组件，滚动更新，监控服务健康。YARN-Service框架主要包括如下组件：1.一个核心框架（A...

qq_20995587

1920人浏览 · 2018-08-31 09:45:02

qq_20995587 · 2018-08-31 09:45:02 发布

YARN-Service

YARN-Service提供一流的支持和API用户托管长期运行的服务。简而言之，它充当容器编排平台管理容器化服务。同时支持Docker容器和传统在YARN之上基于进程的容器。

这个框架的功能包括执行配置解决方案和挂载，生命周期管理，例如停止、启动、删除服务，伸缩服务组件，滚动更新，监控服务健康。

YARN-Service框架主要包括如下组件：

1.一个核心框架（AM）扮演容器编排，负责所有的服务的生命周期管理。

2.RESTful服务器用于用户交互，部署、管理服务。

3.DNS服务依赖YARN Service Registry允许发现服务。

为什么使用YARN Service 框架？

YARN Service框架使得将已有服务迁移到YARN上变的容易。它隐藏了所有复杂的底层细节，包括应用管理，用户不需要去写信的代码。新服务的开发者不需要担心YARN内部机制，只需要专注服务的容器化。

长远来看，这个功能的另一个巨大的胜利是你既可以运行传统批处理程序，也可以运行长期服务，在一个单个平台上。组合这些功能的好处有下面两方面：

1.就像操作单个集群那样操作集群。

2.使得批处理和永久服务可以很好的共享集群资源，提升利用率。

如何使用YARN Service framework部署服务

配置、启动HDFS和YARN组件

像正常一样启动HDFS和YARN组件。启动YARN Service framework，把下列配置添加到yarn-site.xml并且重启RM或者在启动RM之前设置属性。

  <property>
    <description>
      Enable services rest api on ResourceManager.
    </description>
    <name>yarn.webapp.api-service.enable</name>
    <value>true</value>
  </property>

Example service

{
  "name": "sleeper-service",
  "components" : 
    [
      {
        "name": "sleeper",
        "number_of_containers": 1,
        "launch_command": "sleep 900000",
        "resource": {
          "cpus": 1, 
          "memory": "256"
       }
      }
    ]
}

明确运行预构建案例

yarn app -launch <service-name> <example-name>
yarn app -launch my-sleeper sleeper

通过CLI管理services

部署service

yarn app -launch ${SERVICE_NAME} ${PATH_TO_SERVICE_DEF_FILE}

对于当前用户，SERVICE_NAME必须唯一，PATH_TO_SERVICE_DEF_FILE是一个JSON格式的定义文件。

例如：

yarn app -launch sleeper-service /path/to/local/sleeper.json

伸缩服务

yarn app -flex ${SERVICE_NAME} -component ${COMPONENT_NAME} ${NUMBER_OF_CONTAINERS}

yarn app -flex sleeper-service -component sleeper 2

同时支持相对改变，例如+2 or -2

停止服务

停止服务会停止服务的所有容器和AM，但是不会删除服务的状态，例如，在hdfs上的服务根目录。

yarn app -stop ${SERVICE_NAME}

启动服务

yarn app -start ${SERVICE_NAME}

销毁服务

yarn app -destroy ${SERVICE_NAME}

通过API管理服务

如果yarn.webapp.api-service.enable=true，则可以通过API来管理YARN Service

部署一个服务

POST http://localhost:8088/app/v1/services

获取一个服务

GET http://localhost:8088/app/v1/services/${SERVICE_NAME}

伸缩一个服务

PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}/components/${COMPONENT_NAME}

{
    "name": "${COMPONENT_NAME}",
    "number_of_containers": ${COUNT}
}

{
    "name": "sleeper",
    "number_of_containers": 2
}

停止一个服务

PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}

{
  "name": "${SERVICE_NAME}",
  "state": "STOPPED"
}

重启一个服务

PUT http://localhost:8088/app/v1/services/${SERVICE_NAME}

{
  "name": "${SERVICE_NAME}",
  "state": "STARTED"
}

销毁一个服务

DELETE http://localhost:8088/app/v1/services/${SERVICE_NAME}

一个新的service tab会被添加到YARN UI2，服务框架会把数据送到timeline server，然后ui 从timeline server读取

开启timeline server

开启yarn ui ,修改yarn-site.xml，

<property>
    <description>To enable RM web ui2 application.</description>
    <name>yarn.webapp.ui2.enable</name>
    <value>true</value>
  </property>

YARN Service 框架支持kerberized环境，需要指定principal和keytab

{
  "name": "sample-service",
  ...
  ...
  "kerberos_principal" : {
    "principal_name" : "hdfs-demo/_HOST@EXAMPLE.COM",
    "keytab" : "file:///etc/security/keytabs/hdfs.headless.keytab"
  }
}

上述例子，也可以通过Docker实现，只需要改变Artifact类型为DOCKER，id为镜像id即可。

Artifact of a service component，如果不指定，则运行一个裸的命令。

Artifact id : 包的uri，docker的镜像名，服务名

type:docker,tarball。

Component:

服务的一个或者多个组件。

创建一个单组件服务，使用绝大多数属性

POST URL - http://localhost:8088/app/v1/service

{
  "name": "hello-world",
  "version": "1.0.0",
  "description": "hello world example",
  "components" :
    [
      {
        "name": "hello",
        "number_of_containers": 2,
        "artifact": {
          "id": "nginx:latest",
          "type": "DOCKER"
        },
        "launch_command": "./start_nginx.sh",
        "resource": {
          "cpus": 1,
          "memory": "256"
        }
      }
    ]
}

获取信息

GET URL - http://localhost:8088/app/v1/services/hello-worl

{
    "name": "hello-world",
    "version": "1.0.0",
    "description": "hello world example",
    "id": "application_1503963985568_0002",
    "lifetime": -1,
    "state": "STABLE",
    "components": [
        {
            "name": "hello",
            "state": "STABLE",
            "resource": {
                "cpus": 1,
                "memory": "256"
            },
            "configuration": {
                "properties": {},
                "env": {},
                "files": []
            },
            "quicklinks": [],
            "containers": [
                {
                    "id": "container_e03_1503963985568_0002_01_000002",
                    "ip": "10.22.8.143",
                    "hostname": "ctr-e03-1503963985568-0002-01-000002.example.site",
                    "state": "READY",
                    "launch_time": 1504051512412,
                    "bare_host": "host100.cloud.com",
                    "component_instance_name": "hello-0"
                },
                {
                    "id": "container_e03_1503963985568_0002_01_000003",
                    "ip": "10.22.8.144",
                    "hostname": "ctr-e03-1503963985568-0002-01-000003.example.site",
                    "state": "READY",
                    "launch_time": 1504051536450,
                    "bare_host": "host100.cloud.com",
                    "component_instance_name": "hello-1"
                }
            ],
            "launch_command": "./start_nginx.sh",
            "number_of_containers": 1,
            "run_privileged_container": false
        }
    ],
    "configuration": {
        "properties": {},
        "env": {},
        "files": []
    },
    "quicklinks": {}
}

修改生命周期时间

PUT URL - http://localhost:8088/app/v1/services/hello-world

{
  "lifetime": 3600
}

创建一个复杂的HBase服务

POST URL - http://localhost:8088:/app/v1/services/hbase-app-1

{
  "name": "hbase-app-1",
  "version": "1.0.0",
  "description": "hbase service",
  "lifetime": "3600",
  "components": [
    {
      "name": "hbasemaster",
      "number_of_containers": 1,
      "artifact": {
        "id": "hbase:latest",
        "type": "DOCKER"
      },
      "launch_command": "/usr/hdp/current/hbase-master/bin/hbase master start",
      "resource": {
        "cpus": 1,
        "memory": "2048"
      },
      "configuration": {
        "env": {
          "HBASE_LOG_DIR": "<LOG_DIR>"
        },
        "files": [
          {
            "type": "XML",
            "dest_file": "/etc/hadoop/conf/core-site.xml",
            "properties": {
              "fs.defaultFS": "${CLUSTER_FS_URI}"
            }
          },
          {
            "type": "XML",
            "dest_file": "/etc/hbase/conf/hbase-site.xml",
            "properties": {
              "hbase.cluster.distributed": "true",
              "hbase.zookeeper.quorum": "${CLUSTER_ZK_QUORUM}",
              "hbase.rootdir": "${SERVICE_HDFS_DIR}/hbase",
              "zookeeper.znode.parent": "${SERVICE_ZK_PATH}",
              "hbase.master.hostname": "hbasemaster.${SERVICE_NAME}.${USER}.${DOMAIN}",
              "hbase.master.info.port": "16010"
            }
          }
        ]
      }
    },
    {
      "name": "regionserver",
      "number_of_containers": 3,
      "artifact": {
        "id": "hbase:latest",
        "type": "DOCKER"
      },
      "launch_command": "/usr/hdp/current/hbase-regionserver/bin/hbase regionserver start",
      "resource": {
        "cpus": 1,
        "memory": "2048"
      },
      "configuration": {
        "env": {
          "HBASE_LOG_DIR": "<LOG_DIR>"
        },
        "files": [
          {
            "type": "XML",
            "dest_file": "/etc/hadoop/conf/core-site.xml",
            "properties": {
              "fs.defaultFS": "${CLUSTER_FS_URI}"
            }
          },
          {
            "type": "XML",
            "dest_file": "/etc/hbase/conf/hbase-site.xml",
            "properties": {
              "hbase.cluster.distributed": "true",
              "hbase.zookeeper.quorum": "${CLUSTER_ZK_QUORUM}",
              "hbase.rootdir": "${SERVICE_HDFS_DIR}/hbase",
              "zookeeper.znode.parent": "${SERVICE_ZK_PATH}",
              "hbase.master.hostname": "hbasemaster.${SERVICE_NAME}.${USER}.${DOMAIN}",
              "hbase.master.info.port": "16010",
              "hbase.regionserver.hostname": "${COMPONENT_INSTANCE_NAME}.${SERVICE_NAME}.${USER}.${DOMAIN}"
            }
          }
        ]
      }
    }
  ],
  "quicklinks": {
    "HBase Master Status UI": "http://hbasemaster0.${SERVICE_NAME}.${USER}.${DOMAIN}:16010/master-status",
    "Proxied HBase Master Status UI": "http://app-proxy/${DOMAIN}/${USER}/${SERVICE_NAME}/hbasemaster/16010/"
  }
}

Service Discovery

有了DNS支持，用户可以通过定义友好的名称去访问服务

${COMPONENT_INSTANCE_NAME}.${SERVICE_NAME}.${USER}.${DOMAIN}

配置Registry DNS

  <property>
    <description>The domain name for Hadoop cluster associated records.</description>
    <name>hadoop.registry.dns.domain-name</name>
    <value>ycluster</value>
  </property>

  <property>
    <description>The port number for the DNS listener. The default port is 5335.
    If the standard privileged port 53 is used, make sure start the DNS with jsvc support.</description>
    <name>hadoop.registry.dns.bind-port</name>
    <value>5335</value>
  </property>

  <property>
    <description>The DNS functionality is enabled for the cluster. Default is false.</description>
    <name>hadoop.registry.dns.enabled</name>
    <value>true</value>
  </property>

  <property>
    <description>Address associated with the network interface to which the DNS listener should bind.</description>
    <name>hadoop.registry.dns.bind-address</name>
    <value>localhost</value>
  </property>

  <property>
    <description> A comma separated list of hostname:port pairs defining the zookeeper quorum for the YARN registry</description>
    <name>hadoop.registry.zk.quorum</name>
    <value>localhost:2181</value>
  </property>

配置解析网段

 <property>
    <description>The network mask associated with the zone IP range. If specified, it is utilized to ascertain the
    IP range possible and come up with an appropriate reverse zone name.</description>
    <name>hadoop.registry.dns.zone-mask</name>
    <value>255.255.255.0</value>
  </property>

  <property>
    <description>An indicator of the IP range associated with the cluster containers. The setting is utilized for the
     generation of the reverse zone name.</description>
    <name>hadoop.registry.dns.zone-subnet</name>
    <value>172.17.0.0</value>
  </property>

启动Registry DNS Server

yarn --daemon start registrydns

DNS Service提供以下功能：

1.通过DNS暴露存在的服务发现信息

2.容器到IP映射

配置属性

hadoop.registry.zk.quorum:

hadoop.registry.dns.enabled

hadoop.registry.dns.domain-name

hadoop.registry.dns.bind-address

hadoop.registry.dns.bind-port

hadoop.registry.dns.dnssec.enabled

hadoop.registry.dns.private-key-file

hadoop.registry.dns-ttl

hadoop.registry.dns.zone-subnet

hadoop.registry.dns.zone-mask

hadoop.registry.dns.zones-dir

 <property>
    <description>The domain name for Hadoop cluster associated records.</description>
    <name>hadoop.registry.dns.domain-name</name>
    <value>ycluster</value>
  </property>

  <property>
    <description>The port number for the DNS listener. The default port is 5335.
    If the standard privileged port 53 is used, make sure start the DNS with jsvc support.</description>
    <name>hadoop.registry.dns.bind-port</name>
    <value>5335</value>
  </property>

  <property>
    <description>The DNS functionality is enabled for the cluster. Default is false.</description>
    <name>hadoop.registry.dns.enabled</name>
    <value>true</value>
  </property>

  <property>
    <description>Address associated with the network interface to which the DNS listener should bind.</description>
    <name>hadoop.registry.dns.bind-address</name>
    <value>localhost</value>
  </property>

  <property>
    <description>A comma separated list of hostname:port pairs defining the zookeeper quorum for the YARN registry</description>
    <name>hadoop.registry.zk.quorum</name>
    <value>localhost:2181</value>
  </property>

向您推荐>>Eolink开发者社区

权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区

更多推荐

深入理解 Mocha 测试框架：从零实现一个 Mocha

前言什么是自动化测试自动化测试在很多团队中都是Devops环节中很难执行起来的一个环节，主要原因在于测试代码的编写工作很难抽象，99%的场景都需要和业务强绑定，而且写测试代码的编写工作量往往比编写实际业务代码的工作量更多。在一些很多业务场景中投入产出比很低，适合写自动化测试的应该是那些中长期业务以及一些诸如组件一样的基础库。自动化测试是个比较大的概念，其中分类也比较多，比如单元测试，端对端测试，集

云原生

ELK实现containerd的容器日志采集展示【基于logging的全栈监测】

企业级ELK Stack构建介绍

云原生

(20200916 Solved)docker-compose up创建容器自动退出

问题描述如题，创建容器后自动退出了。并且docker start container无效解决方案原因是缺失了控制终端的配置，需要在docker-compose.yml中增加tty:true ，有时候这样也不行，需要再增加一个command:/bin/bash，命令不一定是这个，需要是一个不会退出的命令，然后用-d后台启动容器。Referencesdocker-compose启动容器后自动退出...