docker 应用 —— docker 搭建 hbase 单机环境
文章目录1、系统说明2、创建hbase容器3、进入hbase容器中4、进入 hbase shell 中：5、在win10 中 访问 hbase6、hbase java api1、系统说明我的开发环境是在win10，使用 VMare中搭建centos7的虚拟机。下面的操作就是在 centos7 中使用 docker 搭建 hbase 测试环境 。说明：centos7 ：主机名是 hbase...
·
1、系统说明
我的开发环境是在win10，在 VMware 中使用 centos7的虚拟机。
下面的操作就是在 centos7 中使用 docker 搭建 hbase 单机模式，用于学习和开发使用 。
说明：
centos7 ：主机名是 hbase01
如果让不熟悉大数据的开发人员来 使用zookeeper、hadoop、hbase等组件搭建环境，需要有多台服务器（或者虚拟机）、经历各种的坑，需要几天时间来做， 在实际工作中比较难实现！
2、创建hbase容器
命令：
docker run -d -h hbase01 \
-p 2181:2181 -p 8080:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 \
-p 16000:16000 -p 16010:16010 -p 16201:16201 -p 16301:16301 \
--name hbase6 \
harisekhon/hbase:1.3
参数说明：
-d 表示 后台运行
-h 定义 容器主机 ，我的centos7 主机名是 hbase01
-p 表示 端口映射
--name 表示 容器别名，随机起名，要求惟一
harisekhon/hbase 是 image镜像
执行：
[root@hbase01 ~]# docker run -d -h hbase01 -p 2181:2181 -p 8080:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p 16000:16000 -p 16010:16010 -p 16201:16201 -p 16301:16301 --name hbase6 harisekhon/hbase:1.3
38dd85911fed3a45c16e23a40058117bf67b8bb98bcec6e01a6df6ec07af8597
[root@hbase01 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38dd85911fed harisekhon/hbase:1.3 "/bin/sh -c \"/entryp…" 7 seconds ago Up 5 seconds 0.0.0.0:2181->2181/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8085->8085/tcp, 0.0.0.0:9090->9090/tcp, 0.0.0.0:9095->9095/tcp, 0.0.0.0:16000->16000/tcp, 0.0.0.0:16010->16010/tcp, 0.0.0.0:16201->16201/tcp, 0.0.0.0:16301->16301/tcp hbase6
3、进入hbase容器中
[root@hbase01 ~]# docker exec -it 38dd85911fed /bin/bash
bash-4.4#
4、进入 hbase shell 中：
bash-4.4# hbase shell
hbase(main):001:0> list
TABLE
0 row(s) in 0.3140 seconds
=> []
hbase(main):002:0> list_namespace
list_namespace list_namespace_tables
hbase(main):002:0> list_namespace
NAMESPACE
default
hbase
2 row(s) in 0.0280 seconds
hbase shell 脚本的入门使用： https://blog.csdn.net/xiaojin21cen/article/details/101528834
5、在win10 中 访问 hbase
在win10中配置hosts中，增加映射：
176.16.1.181 hbase01
说明： 176.16.1.181 是虚拟机 centos7的IP
访问 hbase ：
http://hbase01:16010/master-status
6、hbase java api
依赖：
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.3.1</version>
</dependency>
下面代码配置中，除了hbase01 主机名不一致，可能需求修改外，其他的不变。
package hbase;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.Cell;
import org.apache.hadoop.hbase.CellUtil;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.filter.BinaryComparator;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp;
import org.apache.hadoop.hbase.filter.DependentColumnFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.FilterList;
import org.apache.hadoop.hbase.filter.RegexStringComparator;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
import org.apache.hadoop.hbase.filter.SubstringComparator;
import org.apache.hadoop.hbase.filter.ValueFilter;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.Test;
public class HbaseUtility_demo {
public static Configuration conf=null;
public static Connection conn=null;
/**
* 类级别的初始化，只是在类加载的时候做一次 配置zookeeper的端口2181
* 配置zookeeper的仲裁主机名centos，如果有多个机器，主机名间用冒号隔开 配置hbase master
* 还有一种方式是new一个configuration对象，然后用addresource方法去添加xml配置文件 但是像这样显式的配置是会覆盖xml里的配置的
*/
static {
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum","hbase01");
conf.set("hbase.zookeeper.property.clientPort","2181");
conf.set("hbase.master", "hbase01:16000");
conf.setInt("hbase.regionserver.port", 16201);
conf.setInt("hbase.rpc.timeout",200);
conf.setInt("hbase.client.operation.timeout",300);
conf.setInt("hbase.client.scanner.timeout.period",200);
try {
conn = ConnectionFactory.createConnection(conf);
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 建表，建列族
*
* @param tablename,
* @param ColumnFamilys
* NamespaceDescriptor:维护命名空间的信息,但是namespace，一般用shell来建立
* Admin:提供了一个接口来管理 HBase 数据库的表信息
* HTableDescriptor:维护了表的名字及其对应表的列族,通过HTableDescriptor对象设置表的特性
* HColumnDescriptor:维护着关于列族的信息,可以通过HColumnDescriptor对象设置列族的特性
*/
@Test
public void createtable() throws IOException {
String tablename="ns2:student";
String[] ColumnFamilys=new String[]{"data"};
Admin admin = conn.getAdmin();
HTableDescriptor table = new HTableDescriptor(TableName.valueOf(tablename));
for (String family : ColumnFamilys) {
HColumnDescriptor columnfamily = new HColumnDescriptor(family);
table.addFamily(columnfamily);
}
if (admin.tableExists(TableName.valueOf(tablename))) {
System.out.println("Table Exists");
} else {
admin.createTable(table);
System.out.println("Table Created");
admin.close();
}
}
/**
* 插入数据,当指定rowkey已经存在，则会覆盖掉之前的旧数据
*
* @param tablename,
* @param rowkey,
* @param ColumnFamilys,
* @param columns,@values
* Table:用于与单个HBase表通信 Put:用来对单个行执行添加操作
*/
@Test
public void insertdata() throws Exception {
String tablename="ns2:student";
String rowkey="1004";
String ColumnFamilys="data";
String[] columns=new String[]{"name","age","adress"};
String[] values=new String[]{"zhangsan_44","24","guangzhou"};
Table table = conn.getTable(TableName.valueOf(tablename));
Put put = new Put(Bytes.toBytes(rowkey));
for (int i = 0; i < columns.length; i++) {
put.addColumn(Bytes.toBytes(ColumnFamilys), Bytes.toBytes(columns[i]), Bytes.toBytes(values[i]));
}
table.put(put);
System.out.println("data inserted");
table.close();
}
/**
* 扫描全表
*
* @param tablename
*/
@Test
public void scantable() throws IOException {
String tablename="ns2:student";
Scan scan = new Scan();
Table table = conn.getTable(TableName.valueOf(tablename));
ResultScanner rs = table.getScanner(scan);
for (Result result : rs) {
for (Cell cell : result.listCells()) {
System.out.println(Bytes.toString(cell.getRow()) + " " + "column=" + Bytes.toString(cell.getFamily())
+ ":" + Bytes.toString(cell.getQualifier()) + " , value="
+ Bytes.toString(cell.getValue()));
}
System.out.println("");
}
rs.close();
}
/**
* 根据rowkey对表进行scan
*
* @param tablename
* @param rowkey
* scan 'student',{ROWPREFIXFILTER => '1'}
*/
@Test
public void scanrow() throws IOException {
String tablename="ns2:student";
String rowkey="1002";
Get get = new Get(Bytes.toBytes(rowkey));
Table table = conn.getTable(TableName.valueOf(tablename));
Result result = table.get(get);
for (Cell kv : result.listCells()) {
System.out.println(
rowkey + " column=" + Bytes.toString(kv.getFamily()) + ":" + Bytes.toString(kv.getQualifier())
+ "," + " , value=" + Bytes.toString(kv.getValue()));
}
}
/**
* 根据rowkey删除整行的所有列族、所有行、所有版本
*
* @param tablename
* @param rowkey
*/
@Test
public void deleterow() throws IOException {
String tablename="ns2:student";
String rowkey="1001";
Table table = conn.getTable(TableName.valueOf(tablename));
Delete delete = new Delete(Bytes.toBytes(rowkey));
table.delete(delete);
table.close();
System.out.println("row : " + rowkey + " is deleted");
}
/**
* 删除某个row的指定列
*
* @param tablename
* @param rowkey
* @param columnfamily
* @param column
*/
@Test
public void deletecol()throws IOException {
String tablename="ns2:student";
String rowkey="1002";
String columnfamily="data";
String column="age";
Table table = conn.getTable(TableName.valueOf(tablename));
Delete delete = new Delete(Bytes.toBytes(rowkey));
delete.deleteColumn(Bytes.toBytes(columnfamily), Bytes.toBytes(column));
table.delete(delete);
table.close();
System.out.println("row : " + rowkey + " is deleted");
}
/**
* 使用过滤器，获取18-20岁之间的学生信息
*
* @param tablename
* @param age
* @throws IOException
*/
@Test
public void scanfilterage() throws IOException {
String tablename="ns2:student";
String columnfamily="data";
String startage="23";
String endage="23";
System.out.println("scanfilterage()");
SingleColumnValueFilter ft1 = new SingleColumnValueFilter(//
Bytes.toBytes(columnfamily),//
Bytes.toBytes("age"), //
CompareFilter.CompareOp.GREATER, //
Bytes.toBytes(startage)//
);
FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ONE);
filterList.addFilter(ft1);
Scan scan = new Scan();
scan.setFilter(filterList);
Table table = conn.getTable(TableName.valueOf(tablename));
ResultScanner rs = table.getScanner(scan);
for (Result r : rs) {
for (Cell cell : r.listCells()) {
System.out.println(Bytes.toString(cell.getRow()) +"\t column = "
+ Bytes.toString(cell.getFamily()) + ":" + Bytes.toString(cell.getQualifier()) + " value = "
+ Bytes.toString(cell.getValue()));
}
System.out.println("");
}
table.close();
}
@Test
public void testComboFilter() throws IOException {
Connection conn = ConnectionFactory.createConnection(conf);
TableName tname = TableName.valueOf("ns2:student");
Scan scan = new Scan();
//where ... f2:age <= 13
SingleColumnValueFilter ftl = new SingleColumnValueFilter(
Bytes.toBytes("data"),
Bytes.toBytes("age"),
CompareFilter.CompareOp.EQUAL,
// new BinaryComparator(Bytes.toBytes("21"))
Bytes.toBytes("22")
);
FilterList ft = new FilterList(FilterList.Operator.MUST_PASS_ALL);
ft.addFilter(ftl);
scan.setFilter(ft);
Table t = conn.getTable(tname);
ResultScanner rs = t.getScanner(scan);
for (Result result : rs) {
for (Cell cell : result.listCells()) {
System.out.println(Bytes.toString(cell.getRow()) + " " + "column=" + Bytes.toString(cell.getFamily())
+ ":" + Bytes.toString(cell.getQualifier()) + " , value="
+ Bytes.toString(cell.getValue()));
}
System.out.println("");
}
}
/**
* 测试ValueFilter(值过滤器)
* 过滤value的值，含有指定的字符子串
*/
@Test
public void testValueFilter() throws IOException {
Connection conn = ConnectionFactory.createConnection(conf);
TableName tname = TableName.valueOf("ns2:student");
Scan scan = new Scan();
ValueFilter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL, new SubstringComparator("22"));
scan.setFilter(filter);
Table t = conn.getTable(tname);
ResultScanner rs = t.getScanner(scan);
for (Result result : rs) {
for (Cell cell : result.listCells()) {
System.out.println(Bytes.toString(cell.getRow()) + " " + "column=" + Bytes.toString(cell.getFamily())
+ ":" + Bytes.toString(cell.getQualifier()) + " , value="
+ Bytes.toString(cell.getValue()));
}
System.out.println("");
}
}
/**
* DependentColumnFilter 依赖列过滤器 自定义一种过滤基数
*/
@Test
public void testDepFilter() throws IOException {
Connection conn = ConnectionFactory.createConnection(conf);
TableName tname = TableName.valueOf("ns2:student");
Scan scan = new Scan();
DependentColumnFilter filter = new DependentColumnFilter(//
Bytes.toBytes("data"),//
Bytes.toBytes("name"),//
true,//
CompareFilter.CompareOp.NOT_EQUAL,//
new BinaryComparator(Bytes.toBytes("22"))//
);
scan.setFilter(filter);
Table t = conn.getTable(tname);
ResultScanner rs = t.getScanner(scan);
for (Result result : rs) {
for (Cell cell : result.listCells()) {
System.out.println(Bytes.toString(cell.getRow()) + " " + "column=" + Bytes.toString(cell.getFamily())
+ ":" + Bytes.toString(cell.getQualifier()) + " , value="
+ Bytes.toString(cell.getValue()));
}
System.out.println("");
}
}
@Test //扫描数据
public void getRow(String tableName, String rowKey) throws IOException{
TableName tname = TableName.valueOf("ns2:student");
Scan scan = new Scan();
Table t = conn.getTable(tname);
ResultScanner rs = t.getScanner(scan);
for (Result result : rs) {
for(Cell cell : result.rawCells()){
System.out.println("行键:" + Bytes.toString(result.getRow()));
System.out.println("列族" + Bytes.toString(CellUtil.cloneFamily(cell)));
System.out.println("列:" + Bytes.toString(CellUtil.cloneQualifier(cell)));
System.out.println("值:" + Bytes.toString(CellUtil.cloneValue(cell)));
System.out.println("时间戳:" + cell.getTimestamp());
}
}
}
public static void main(String[] args) throws IOException {
// createtable("student", "imformation", "score");
//
// insertdata("student", "1", "imformation", col1, val1);
// insertdata("student", "1", "imformation", col2, val2);
//
// deleterow("student", "1");
// deletecol("student", "1", "imformation", "chinese");
// deleteversion("student", "1", "imformation", 1533482642629L);
// deletefamily("student", "imformation");
// droptable("student");
//
// scantable("student");
// scanrow("student", "1");
// scanspecifycolumn("student", "1", "imformation", "chinese");
// scanspecifytimestamp("student", "imformation", 1533482642629L);
// scanallversion("student", "1");
// scanfilterage("student", 18, 20);
}
}
权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区
更多推荐
3
0- 0
已为社区贡献14条内容
所有评论(0)