各应用程序及版本

  • storm1.1.1
  • zookeeper3.4.10

拓朴结构

  • nimbus、ui：CentOS65App
  • supervisor：CentOS65M1、CentOS65M2、CentOS65M3

配置文件

  • zoo.cfg，zookeeper仅安装在三台supervisor服务器

server.6=CentOS65M1:2888:3888
server.7=CentOS65M2:2888:3888
server.8=CentOS65M3:2888:3888
  • storm.yaml，在四台服务器配置一样

storm.zookeeper.servers:
     - "CentOS65M1"
     - "CentOS65M2"
     - "CentOS65M3"

nimbus.seeds: ["CentOS65App"]

storm.zookeeper.port: 2181
storm.local.dir: "/app/data"
ui.port: 9099

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

按序启动zookeeper，nimbus，ui，supervisor

storm ui可以在浏览器正常访问

http://centos65app:9099/index.html

不部署topology，集群一切正常；一旦部署了topology，supervisor就抛异常了。

./bin/storm jar /opt/storm_jars/storm-starter-1.1.1.jar org.apache.storm.starter.WordCountTopology WordCountTopology
过一两分钟，从storm ui就发现Supervisors由3变为0。逐个进入三台supervisor服务器，jps后，发现确实原来的supervisor已经不存在了。在supervisor服务器的[storm_home]/logs/supervisor.log文件发现有如下异常： 

2018-01-17 11:18:45.329 o.a.s.d.s.Slot main [WARN] SLOT CentOS65M1:6700 Starting in state EMPTY - assignment null
2018-01-17 11:18:45.330 o.a.s.d.s.Slot main [WARN] SLOT CentOS65M1:6701 Starting in state EMPTY - assignment null
2018-01-17 11:18:45.331 o.a.s.d.s.Slot main [WARN] SLOT CentOS65M1:6702 Starting in state EMPTY - assignment null
2018-01-17 11:18:45.331 o.a.s.d.s.Slot main [WARN] SLOT CentOS65M1:6703 Starting in state EMPTY - assignment null
2018-01-17 11:18:45.337 o.a.s.l.AsyncLocalizer main [INFO] Cleaning up unused topologies in /app/data/supervisor/stormdist
2018-01-17 11:18:45.344 o.a.s.d.s.Supervisor main [INFO] Starting supervisor with id 7cb2d4a8-a09c-4bb3-9449-16cb1d6b96da at host CentOS65M1.
2018-01-17 11:18:45.354 o.a.s.d.m.MetricsUtils main [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
2018-01-17 11:18:45.356 o.a.s.d.m.r.JmxPreparableReporter main [INFO] Preparing...
2018-01-17 11:18:45.368 o.a.s.m.StormMetricsRegistry main [INFO] Started statistics report plugin...
2018-01-17 11:18:47.359 o.a.s.d.s.Slot SLOT_6701 [INFO] STATE EMPTY msInState: 2034 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 6
2018-01-17 11:18:47.360 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE EMPTY msInState: 2031 -> WAITING_FOR_BASIC_LOCALIZATION msInState: 0
2018-01-17 11:18:47.476 o.a.s.u.StormBoundedExponentialBackoffRetry Async Localizer [WARN] WILL SLEEP FOR 2001ms (NOT MAX)
2018-01-17 11:18:49.480 o.a.s.u.StormBoundedExponentialBackoffRetry Async Localizer [WARN] WILL SLEEP FOR 2002ms (NOT MAX)
2018-01-17 11:18:51.485 o.a.s.u.StormBoundedExponentialBackoffRetry Async Localizer [WARN] WILL SLEEP FOR 2006ms (NOT MAX)
2018-01-17 11:18:53.493 o.a.s.u.StormBoundedExponentialBackoffRetry Async Localizer [WARN] WILL SLEEP FOR 2015ms (NOT MAX)
2018-01-17 11:18:55.510 o.a.s.u.StormBoundedExponentialBackoffRetry Async Localizer [WARN] WILL SLEEP FOR 2024ms (NOT MAX)
2018-01-17 11:18:57.537 o.a.s.u.NimbusClient Async Localizer [WARN] Ignoring exception while trying to get leader nimbus info from CentOS65App. will retry with a different seed host.
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.NoRouteToHostException: 没有到主机的路由 (Host unreachable)
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:108) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:127) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:83) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:57) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.Utils.getClientBlobStoreForSupervisor(Utils.java:538) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:121) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101) ~[storm-core-1.1.1.jar:1.1.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.NoRouteToHostException: 没有到主机的路由 (Host unreachable)
        at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:100) ~[storm-core-1.1.1.jar:1.1.1]
        ... 13 more
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.NoRouteToHostException: 没有到主机的路由 (Host unreachable)
        at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:105) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:100) ~[storm-core-1.1.1.jar:1.1.1]
        ... 13 more
Caused by: java.net.NoRouteToHostException: 没有到主机的路由 (Host unreachable)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_121]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_121]
        at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_121]
        at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:105) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:100) ~[storm-core-1.1.1.jar:1.1.1]
        ... 13 more
2018-01-17 11:18:57.537 o.a.s.l.AsyncLocalizer Async Localizer [WARN] Failed to download basic resources for topology-id WordCountTopology-1-1516157915
2018-01-17 11:18:57.537 o.a.s.d.s.AdvancedFSOps Async Localizer [INFO] Deleting path /app/data/supervisor/tmp/27e1b9d2-9993-43d1-8d1d-9156b96a8bd2
2018-01-17 11:18:57.543 o.a.s.d.s.AdvancedFSOps Async Localizer [INFO] Deleting path /app/data/supervisor/stormdist/WordCountTopology-1-1516157915
2018-01-17 11:18:57.543 o.a.s.l.AsyncLocalizer Async Localizer [WARN] Caught Exception While Downloading (rethrowing)...
org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [CentOS65App]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
        at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:111) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:57) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.Utils.getClientBlobStoreForSupervisor(Utils.java:538) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:121) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101) ~[storm-core-1.1.1.jar:1.1.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2018-01-17 11:18:57.544 o.a.s.d.s.Slot SLOT_6700 [ERROR] Error when processing event
java.util.concurrent.ExecutionException: org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [CentOS65App]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_121]
        at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_121]
        at org.apache.storm.localizer.LocalDownloadedResource$NoCancelFuture.get(LocalDownloadedResource.java:63) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBasicLocalization(Slot.java:413) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:273) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:741) ~[storm-core-1.1.1.jar:1.1.1]
Caused by: org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [CentOS65App]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
        at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:111) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:57) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.Utils.getClientBlobStoreForSupervisor(Utils.java:538) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:121) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101) ~[storm-core-1.1.1.jar:1.1.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2018-01-17 11:18:57.544 o.a.s.u.Utils SLOT_6700 [ERROR] Halting process: Error when processing an event
java.lang.RuntimeException: Halting process: Error when processing an event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:1773) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:774) ~[storm-core-1.1.1.jar:1.1.1]
2018-01-17 11:18:57.544 o.a.s.d.s.Slot SLOT_6701 [ERROR] Error when processing event
java.util.concurrent.ExecutionException: org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [CentOS65App]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_121]
        at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_121]
        at org.apache.storm.localizer.LocalDownloadedResource$NoCancelFuture.get(LocalDownloadedResource.java:63) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.handleWaitingForBasicLocalization(Slot.java:413) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.stateMachineStep(Slot.java:273) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:741) ~[storm-core-1.1.1.jar:1.1.1]
Caused by: org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [CentOS65App]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
        at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:111) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:57) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:268) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.utils.Utils.getClientBlobStoreForSupervisor(Utils.java:538) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:121) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101) ~[storm-core-1.1.1.jar:1.1.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2018-01-17 11:18:57.544 o.a.s.u.Utils SLOT_6701 [ERROR] Halting process: Error when processing an event
java.lang.RuntimeException: Halting process: Error when processing an event
        at org.apache.storm.utils.Utils.exitProcess(Utils.java:1773) ~[storm-core-1.1.1.jar:1.1.1]
        at org.apache.storm.daemon.supervisor.Slot.run(Slot.java:774) ~[storm-core-1.1.1.jar:1.1.1]
2018-01-17 11:18:57.562 o.a.s.d.s.Supervisor Thread-5 [INFO] Shutting down supervisor 7cb2d4a8-a09c-4bb3-9449-16cb1d6b96da
2018-01-17 11:18:57.565 o.a.s.e.EventManagerImp Thread-4 [INFO] Event manager interrupted
解决办法有以下四种，根据情况依次偿试：
  • ping nimbus主机，如果不通，则检查这四台服务器的/etc/hosts（之前公司ip和家用ip共同配置在hosts，改为单一ip）
  • 在zookeeper中，删除storm目录
./bin/zkCli.sh
ls /
rmr /storm
  • 关闭防火墙
  • 防火墙中，只打开需要的端口
登录storm ui，在Nimbus Summary中看到nimbus用到的端口号为6627，在防火墙中打开此端口（之前已经打开了9099、2181、6700等端口，nimbus端口要单独打开）
# 大数据 # 集群
Logo
向您推荐>>Eolink开发者社区

权威｜前沿｜技术｜干货｜国内首个API全生命周期开发者社区

更多推荐

深入理解 Mocha 测试框架：从零实现一个 Mocha

前言什么是自动化测试自动化测试在很多团队中都是Devops环节中很难执行起来的一个环节，主要原因在于测试代码的编写工作很难抽象，99%的场景都需要和业务强绑定，而且写测试代码的编写工作量往往比编写实际业务代码的工作量更多。在一些很多业务场景中投入产出比很低，适合写自动化测试的应该是那些中长期业务以及一些诸如组件一样的基础库。自动化测试是个比较大的概念，其中分类也比较多，比如单元测试，端对端测试，集

avatar 云原生

ELK实现containerd的容器日志采集展示【基于logging的全栈监测】

企业级ELK Stack构建介绍

avatar 云原生

(20200916 Solved)docker-compose up创建容器自动退出

问题描述如题，创建容器后自动退出了。并且docker start container无效解决方案原因是缺失了控制终端的配置，需要在docker-compose.yml中增加tty:true ，有时候这样也不行，需要再增加一个command:/bin/bash，命令不一定是这个，需要是一个不会退出的命令，然后用-d后台启动容器。Referencesdocker-compose启动容器后自动退出...

avatar 云原生