storm 远程提交任务
远程提交任务还是十分必要的,如果有一个storm集群用于调试(有点奢侈,但是必要),可以从本地开发机器上直接提交topology. 在生产环境下,通常我们通过Jenkins编译代码,然后提交到storm集群,这也是远程提交的一种方式。这样,开发团队不会直接和storm线上环境打交道,保证了集群的安全性。基本原理是,通过运行本地的storm.jar和远程storm集群的nimbus通信,将任务(to
·
远程提交任务还是十分必要的,如果有一个storm集群用于调试(有点奢侈,但是必要),可以从本地开发机器上直接提交topology. 在生产环境下,通常我们通过Jenkins编译代码,然后提交到storm集群,这也是远程提交的一种方式。这样,开发团队不会直接和storm线上环境打交道,保证了集群的安全性。
基本原理是,通过运行本地的storm.jar和远程storm集群的nimbus通信,将任务(topology的jar包)提交到nimbus并运行。下面是步骤:
下载storm release包
下载方式是
wget http://mirror.bit.edu.cn/apache/storm/apache-storm-0.9.5/apache-storm-0.9.5.tar.gz
然后设置PATH环境变量
export PATH=${your-storm-folder}/apache-storm-0.9.5/bin:$PATH
设置nimbus主机地址
创建~/.storm/storm.yaml文件
添加配置
nimbus.host: "192.168.11.0"
运行storm程序测试
$ storm list
Running: /home/dean/Downloads/jdk1.7.0_51/bin/java -client -Dstorm.options= -Dstorm.home=/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5 -Dstorm.log.dir=/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/math.numeric-tower-0.0.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/jgrapht-core-0.9.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/carbonite-1.4.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-codec-1.6.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/servlet-api-2.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/joda-time-2.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/ring-devel-0.3.11.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/logback-classic-1.0.13.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/kryo-2.21.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/clj-time-0.4.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/ring-core-1.1.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-logging-1.1.3.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/logback-core-1.0.13.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/storm-core-0.9.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/snakeyaml-1.11.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/reflectasm-1.07-shaded.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/hiccup-0.3.6.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/objenesis-1.2.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/tools.cli-0.2.4.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/asm-4.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-fileupload-1.2.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/minlog-1.2.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/clojure-1.5.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/jetty-util-6.1.26.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/log4j-over-slf4j-1.6.6.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-lang-2.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/clj-stacktrace-0.2.2.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/jetty-6.1.26.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-io-2.4.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/commons-exec-1.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/json-simple-1.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/compojure-1.1.3.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/jline-2.11.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/chill-java-0.3.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/ring-servlet-0.3.11.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/core.incubator-0.1.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/clout-1.0.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/ring-jetty-adapter-0.3.11.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/disruptor-2.10.1.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/slf4j-api-1.7.5.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/tools.macro-0.1.0.jar:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/lib/tools.logging-0.2.3.jar:/home/dean/.storm:/home/dean/work/gitlab/nj/letv/study/storm/apache-storm-0.9.5/bin backtype.storm.command.list
1242 [main] INFO backtype.storm.thrift - Connecting to Nimbus at ${nimbus-host}:6627
No topologies running.
部署成功。
提交topology
语法 storm jar your-jar your-main-class, 比如:
storm jar target/uberjar/kafka2hdfs-0.1.0-SNAPSHOT-standalone.jar kafka2hdfs.core
这样就能够通过本地~/.storm/storm.yaml的设置提交到storm集群
更多推荐
已为社区贡献30条内容
所有评论(0)