平台:H3C DataEngine

1.上传文件到linux文件系统

[root@node1 mydata]# ll
total 129648
-rw-r--r--. 1 root root 120734753 Dec  6 12:53 bigtable.bigtable
-rw-r--r--. 1 root root  12018355 Dec  6 12:51 smalltable.smalltable

2.加载数据到hive

创建表

create table bigtable(
	id int,
	time string,
	sessionId string,
	key string,
	up int,
	down int,
	url string) 
row format delimited fields terminated by '\t';

试图使用load命令加载本地文件到hive中(环境是使用beeline连接hive,因为涉及hdfs的操作,所以登录用户必须具备hdfs操作权限,因此选择使用hdfs用户登录hive)

0: jdbc:hive2://node1:10000/> load data local inpath '/usr/hdp/mydata/bigtable.bigtable' into table bigtable;
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: 
Principal [name=hdfs, type=US  ER] does not have following privileges for operation 
LOAD [[INSERT, DELETE, OBJECT OWNERSHIP] on Object [type=LOCAL_URI, na  me=file:/usr/hdp/mydata/bigtable.bigtable]] (state=42000,code=40000)

报权限问题,给出的提示是[INSERT, DELETE, OBJECT OWNERSHIP],首先是[INSERT, DELETE]权限问题,该问题是由于hdfs用户对该文件的权限不足,上面看到只有可读权限,因此提高文件的权限

[root@node1 mydata]# ll
total 129648
-rw-rw-rw-. 1 root root 120734753 Dec  6 12:53 bigtable.bigtable
-rw-rw-rw-. 1 root root  12018355 Dec  6 12:51 smalltable.smalltable

在此加载,发现[INSERT, DELETE]权限问题已经解决,但[OBJECT OWNERSHIP]权限问题依然存在

0: jdbc:hive2://node1:10000/> load data local inpath '/usr/hdp/mydata/bigtable.bigtable' into table bigtable;
Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: 
Principal [name=hdfs, type=USER] does not have following privileges for operation 
LOAD [[OBJECT OWNERSHIP] on Object [type=LOCAL_URI, name=file:/usr/hdp  /mydata/bigtable.bigtable]] (state=42000,code=40000)

这个问题就很显而易见了,使用的是hdfs用户登录,但文件的所有者属于root,因此修改文件的所有者再次导入

[root@node1 mydata]# ll
total 129648
-rw-rw-rw-. 1 hdfs root 120734753 Dec  6 12:53 bigtable.bigtable
-rw-rw-rw-. 1 hdfs root  12018355 Dec  6 12:51 smalltable.smalltable

问题解决

0: jdbc:hive2://node1:10000/> load data local inpath '/usr/hdp/mydata/bigtable.bigtable' into table bigtable;
INFO  : Loading data to table hc.bigtable from file:/usr/hdp/mydata/bigtable.bigtable
INFO  : Table hc.bigtable stats: [numFiles=1, totalSize=120734753]
No rows affected (6.948 seconds)

检验数据是否导入

0: jdbc:hive2://node1:10000/> select url from bigtable limit 10;
+----------------------------------------------------------------+--+
|                              url                               |
+----------------------------------------------------------------+--+
| http://www.qiyi.com/                                           |
| http://www.booksky.org/BookDetail.aspx?BookID=1050804&Level=1  |
| http://www.bblianmeng.com/                                     |
| http://lib.scnu.edu.cn/                                        |
| http://proxyie.cn/                                             |
| http://movie.douban.com/review/1128960/                        |
| http://www.youku.com/                                          |
| http://hf.house365.com/                                        |
| http://www.kz321.com/                                          |
| http://www.iqshuma.com/                                        |
+----------------------------------------------------------------+--+
10 rows selected (0.55 seconds)

同理加载hdfs上的数据包权限异常同样这样处理。

Logo

更多推荐