Could not connect to ZooKeeper slave01:2181,slave02:2181,slave03:2181
这是在通过写MapReduce读取HBase的数据，给指定的列加上索引到solr发现的zookeeper.out日志：2015-11-20 16:59:20,603 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from
·
这是在通过写MapReduce读取HBase的数据，给指定的列加上索引到solr发现的
zookeeper.out日志：
2015-11-20 16:59:20,603 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.77 - max is 602015-11-20 16:59:20,619 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.76 - max is 60
2015-11-20 16:59:20,762 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.76 - max is 60
2015-11-20 16:59:20,774 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.77 - max is 60
2015-11-20 16:59:20,915 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.76 - max is 60
2015-11-20 16:59:20,973 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.77 - max is 60
2015-11-20 16:59:21,219 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.77 - max is 60
2015-11-20 16:59:22,058 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /10.21.3.77 - max is 60
MapReduce执行报错：
15/11/20 16:59:41 INFO mapreduce.Job: Task Id : attempt_1448000031899_0003_m_000012_2, Status : FAILED
Error: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper slave01:2181,slave02:2181,slave03:2181 within 10000 ms
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:164)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:212)
at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:241)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:524)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at com.yiban.mapreduce.HBaseToSolrIndex$HBaseToSolrIndexMapper.map(HBaseToSolrIndex.java:60)
at com.yiban.mapreduce.HBaseToSolrIndex$HBaseToSolrIndexMapper.map(HBaseToSolrIndex.java:29)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper slave01:2181,slave02:2181,slave03:2181 within 10000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:156)
... 18 more
查看zookeeper 2181端口的连接数：使用netstat -nap | grep 2181 | wc -l查看
正常连接数
[root@slave01 hbase-1.0.2]# netstat -nap | grep 2181 | wc -l
11
[root@slave02 lib]# netstat -nap | grep 2181 | wc -l
11
[root@slave03 lib]# netstat -nap | grep 2181 | wc -l
15
运行MapReduce连接数，太多不正常
[root@slave01 hbase-1.0.2]# netstat -nap | grep 2181 | wc -l
801
[root@slave01 hbase-1.0.2]# netstat -nap | grep 2181 | wc -l
826
[root@slave02 lib]# netstat -nap | grep 2181 | wc -l
566
[root@slave02 lib]# netstat -nap | grep 2181 | wc -l
641
[root@slave03 lib]# netstat -nap | grep 2181 | wc -l
196
[root@slave03 lib]# netstat -nap | grep 2181 | wc -l
529
[root@slave03 lib]# netstat -nap | grep 2181 | wc -l
563
[root@slave03 lib]# ^C
[root@slave03 lib]# netstat -nap | grep 2181 | wc -l
546
有问题代码：
写在MapReduce里的map方法里面(这里是局部变量，每调用一次map方法就会创建一个zookeeper连接，导致连接数直线上升)
CloudSolrServer solrServer = new CloudSolrServer("slave01:2181,slave02:2181,slave03:2181");//获取zookeeper地址
正常代码：
写在加载时的静态块里面(整个MapReduce只创建一个zookeeper连接)
private static CloudSolrServer solrServer;
static{
solrServer = new CloudSolrServer("slave01:2181,slave02:2181,slave03:2181");//获取zookeeper地址
}
总结：不管是什么连接，能用全局创建尽量用全局创建，这样能避免连接数过多问题
权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区
更多推荐
0
0- 0
已为社区贡献1条内容
所有评论(0)