CDH安装中遇到的问题解决
--------------------------cdh中重装hbase后之前的脚本残留解决---------------------1.切换到zookeeper的bin目录; //zookeeper-client status命令可以找到该bin目录2.执行$sh zkCli.sh输入 ls /4.输入rmr /hbas
·
--------------------------cdh中重装hbase后之前的脚本残留解决---------------------
1.切换到zookeeper的bin目录;
//zookeeper-client status命令可以找到该bin目录
2.执行$sh zkCli.sh
输入 ls /
4.输入rmr /hbase
5.退出重启即可
-------------------------cdh重装过程中有datanode启动报错-----------------------------------
there are 2 Possible Solutions to resolve
First:
Your namenode and datanode cluster ID does not match, make sure to make them the same.
In name node, change ur cluster id in the file located in:
$ nano HADOOP_FILE_SYSTEM/namenode/current/VERSION
In data node you cluster id is stored in the file:
$ nano HADOOP_FILE_SYSTEM/datanode/current/VERSION
-------------------------cdh使用过程中kafka崩溃后重装清理残留的kafka步骤-----------------------------------
1.kafka-topics.sh --delete --zookeeper
提示marked for deletion
并没有真正删除,如果要真正删除
配置delete.topic.enable=true
配置文件在kafka\config目录server.properties
2、删除kafka存储目录(server.propertiewenjian log.dirs配置,默认为“/tmp/kafka-logs”)下对应的topic。(不同broker下存储的topic不一定相同,所有broker都要看一下)
3、进入zookeeper客户端删掉对应topic
zkCli.sh -server 127.0.0.1:42182
找到topic目录
ls /brokers/topics
删掉对应topic
rmr /brokers/topic/topic-name
找到目录
ls /config/topics
删掉对应topic
rmr /config/topics/topic-name
-------------------------cdh集群中spark-shell
启动报错-----------------------------------
[root@archive bin]# ./spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:117)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:117)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:117)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:103)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 7 more
在CDH集群中启动spark-shell是默认以yarn模式启动的,缺少环境变量的配置。
1.如果要在本地模式启动可以加参数:spark-shell --master local[4]
2.如果要在集群模式启动的话,需要配置spark的环境变量
spark-conf/spark-env.sh 的 Spark 客户端高级配置代码段(安全阀)
<strong><span style="color:#ff0000;">export SPARK_DIST_CLASSPATH=$SPARK_DIST_CLASSPATH:/opt/cloudera/parcels/CDH/jars/htrace-core-3.1.0-incubating.jar</span></strong>
:/etc/hbase/conf:/usr/local/jars/elasticsearch-spark_2.10-2.2.0.jar:/usr/local/jars/joda-time-2.9.1.jar:/usr/local/jars/es-shaded-1.0-SNAPSHOT.jar
更多推荐
已为社区贡献1条内容
所有评论(0)