Hive高可用配置
Hive 高可用1. HiveServer2高可用及Metastore高可用使用Zookeeper实现了HiveServer2的HA功能(ZooKeeper Service Discovery),Client端可以通过指定一个nameSpace来连接HiveServer2,而不是指定某一个host和portMetastore高可用原理:Hive Metastore客户端始终使用第一个url连接到m
Hive 高可用
1. HiveServer2高可用及Metastore高可用
使用Zookeeper实现了HiveServer2的HA功能(ZooKeeper Service Discovery),Client端可以通过指定一个nameSpace来连接HiveServer2,而不是指定某一个host和port
Metastore高可用原理:
Hive Metastore客户端始终使用第一个url连接到metastore服务器,如果Metastore服务器变得无法访问,则客户端从列表中随机选取一个url并尝试与之建立连接
在node1和node3上分别启用两个HiveServer2的实例,并通过zookeeper完成HA的配置。
1.1 node01节点修改hive-site.xml配置
<!--配置hiveserver2高可用-->
<property>
<name>hive.server2.support.dynamic.service.discovery</name>
<value>true</value>
</property>
<property>
<name>hive.server2.zookeeper.namespace</name>
<value>hiveserver2_zk</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value> node01:2181,node02:2181,node03:2181</value>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>node01</value>
</property>
<!--配置metastore高可用-->
<property>
<name>hive.metastore.uris</name>
<value>thrift://node01:9083,thrift://node03:9083</value>
</property>
1.2 node03节点同步配置及信息修改
将安装好的hive文件夹同步到node03节点上,修改node03上的hive-site.xml配置如下:
<property>
<name>hive.server2.thrift.bind.host</name>
<value>node03</value>
</property>
1.3 重启服务
分别重启node01,node03节点上的的hiveServer2和metaStore服务
nohup hive --service hiveserver2 >> ./hiveserver.log 2>&1 &
nohup hive --service metastore >> ./metastore.log 2>&1 &
1.4 在Zookeeper中检查配置
修改完配置后,可通过zookeeper_client命令进行查看,hiveserver2在zookeeper中是否注册成功
[zk: localhost:2181(CONNECTED) 1] ls /hiveserver2_zk
1.5 测试验证
1.5.1 验证HiveServer2/mestastore是否是高可用
在node03节点上,杀掉占用10000端口的进程,即杀掉node03的hiveServer2进程
[root@node03 logs]$ netstat -ntpl |grep 10000
node03节点杀死mestastore进程
[root@node03 ~]# ps -ef | grep metastore
kill -9 pid
1.5.2 beeline测试连接
!connect jdbc:hive2://cm1:2181,cm2:2181,cm3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk 用户名 密码
[root@node01 ~]# beeline
beeline> !connect jdbc:hive2://cm1:2181,cm2:2181,cm3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk robin 123456
0: jdbc:hive2://cm1:2181,cm2:2181,cm3:2181/> show databases;
参数含义说明:
jdbc:hive2://<zookeeper quorum>/<dbName>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
// 参数含义
<zookeeper quorum> 为Zookeeper的集群链接串,如node1:2181,node2:2181,node3:2181
<dbName> 为Hive数据库,默认为default
serviceDiscoveryMode=zooKeeper 指定模式为zooKeeper
zooKeeperNamespace=hiveserver2 指定ZK中的nameSpace,即参数hive.server2.zookeeper.namespace所定义的
至此, HiveServer2及Metastore的多实例高可用Ha配置完成
1.5.3 Java API测试:
1)添加maven依赖
<dependencies>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>3.1.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>3.1.3</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hive/hive-metastore -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-metastore</artifactId>
<version>3.1.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hive/hive-metastore -->
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>3.1.2</version>
</dependency>
</dependencies>
2)代码示例:
package com.jd.test;
import java.sql.*;
public class HiveOption {
private static String driverName = "org.apache.hive.jdbc.HiveDriver";
public static void main(String[] args) throws SQLException {
try {
Class.forName(driverName);
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
Connection conn = DriverManager.getConnection("jdbc:hive2://cm1:2181,cm2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk", "robin", "123456");
Statement stmt = conn.createStatement();
//创建数据库
String sql = "create database db1";
stmt.execute(sql);
// 查询所有数据库
sql = "show databases";
ResultSet res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(res.getString(1));
}
//查询数据
sql = "select * from db_hive.stu";
res = stmt.executeQuery(sql);
while (res.next()) {
System.out.println(res.getString(1)+"-"+res.getString(2));
}
}
}
3)连接HiveServer2修改hive配置的方法:
使用state.execute 直接执行set 操作
stmt.execute("set hive.execution.engine=spark");
通过连接属性设置
Properties prop = new Properties();
prop.setProperty("user","robin");
prop.setProperty("password","123456");
prop.setProperty("hive.execution.engine","spark");
Connection conn = DriverManager.getConnection("jdbc:hive2://cm1:2181,cm2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk", prop);
Statement stmt = conn.createStatement();
更多推荐
所有评论(0)