问题场景

在linux上,使用hadoop jar命令提交MR程序

 hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount -Dmapreduce.job.queuename=default /user/input /output1

结果遇到下列错误

java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1662088449147_0006 to YARN : User sfapp cannot submit applications to queue root.sfapp
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:306)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1662088449147_0006 to YARN : User sfapp cannot submit applications to queue root.sfapp
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:270)
        at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:290)
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290)

问题原因

根据关键字cannot submit applications to queue在源码里搜索,发现是如下代码抛出来的,表示该用户没有该队列的权限。
在这里插入图片描述
但是在fair-scheduler.xml文件中,关于default队列的权限配置,是不区分用户权限的
在这里插入图片描述
进一步分析得知,提交时指定队列名为-Dmapreduce.job.queuename=default(实际应该写成-Dmapreduce.job.queuename=root.default),由于不是写成root.default而是default,导致yarn源码的assignToQueue方法没有获取到匹配的队列,于是根据自身逻辑返回了root.sfapp队列
在这里插入图片描述
很明显,并没有root.sfapp队列,因此在其后的权限校验时,就抛出了开头的异常信息
在这里插入图片描述

问题解决

提交命令的队列名改为root.default

 hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount -Dmapreduce.job.queuename=root.default /user/input /output1
Logo

为开发者提供学习成长、分享交流、生态实践、资源工具等服务,帮助开发者快速成长。

更多推荐