Failed to submit application_1662088449147_0004 to YARN : User sfapp cannot submit applications to q
提交MR任务,报无队列权限问题
·
问题场景
在linux上,使用hadoop jar命令提交MR程序
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount -Dmapreduce.job.queuename=default /user/input /output1
结果遇到下列错误
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1662088449147_0006 to YARN : User sfapp cannot submit applications to queue root.sfapp
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:306)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1662088449147_0006 to YARN : User sfapp cannot submit applications to queue root.sfapp
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:270)
at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:290)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290)
问题原因
根据关键字cannot submit applications to queue
在源码里搜索,发现是如下代码抛出来的,表示该用户没有该队列的权限。
但是在fair-scheduler.xml文件中,关于default队列的权限配置,是不区分用户权限的
进一步分析得知,提交时指定队列名为-Dmapreduce.job.queuename=default
(实际应该写成-Dmapreduce.job.queuename=root.default
),由于不是写成root.default
而是default
,导致yarn源码的assignToQueue
方法没有获取到匹配的队列,于是根据自身逻辑返回了root.sfapp
队列
很明显,并没有root.sfapp
队列,因此在其后的权限校验时,就抛出了开头的异常信息
问题解决
提交命令的队列名改为root.default
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount -Dmapreduce.job.queuename=root.default /user/input /output1
更多推荐
已为社区贡献2条内容
所有评论(0)