有兴趣想学习国内整套Spark+Spark Streaming+Machine learning最顶级课程的,可加我qq  471186150。共享视频,性价比超高!



明:

  • 目前安装了4个Redhat linux操作系统,主机名分别为hadoop01,hadoop02,hadoop03,hadoop04
  • 其中hadoop01为主节点,hadoop02,hadoop03,hadoop04为从节点
  • 四个节点ip地址为192.168.10.61~64.
  • 如果不修改hosts文件,从第二部开始可将hadoop01~04改为所对应的ip地址,如第4步中的scp ~/.ssh/id_rsa.pub root@hadoop02:~命令可改为scp ~/.ssh/id_rsa.pub root@192.168.10.62:~

第1步:hosts文件修改(可选)

如果将来想采用ssh 主机名的方式来访问其余节点,请确保各个节点上的hosts文件下有对各个ip以及主机名的映射,可通过如下命令进行查看

<code class="language-shell hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat /etc/hosts</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

hosts 
如果没有将ip地址与主机名相对应,请按照如上所示,修改hosts文件

<code class="language-shell hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vim /etc/hosts</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

第2步:生成密钥对

每个节点上执行如下命令来生成密钥对,一直点击回车即可

<code class="language-shell hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ssh<span class="hljs-attribute" style="box-sizing: border-box;">-keygen</span> <span class="hljs-attribute" style="box-sizing: border-box;">-t</span> rsa <span class="hljs-attribute" style="box-sizing: border-box;">-P</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

生成密钥对

第3步:本机ssh访问

每个节点上执行如下命令将公钥文件写入授权文件中,并赋值权限

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat ~/<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ssh</span>/id_rsa<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.pub</span> >> ~/<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ssh</span>/authorized_keys
chmod <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">600</span> ~/<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ssh</span>/authorized_keys</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

在每个节点上可通过输入两次ssh 本节点主机名命令来验证(注意,第一次输入ssh 本节点主机名会询问,输入yes,以后就不需要输入密码了) 
ssh

第4步:主节点到从节点无密码ssh访问

主节点分别执行如下命令将主节点的公钥文件拷贝到从节点上(注意,当进行远程拷贝的时候会要求输入节点的密码)

<code class="language-shell hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">scp ~<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/.ssh/id</span>_rsa.pub  root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@hadoop02</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:~</span>
scp ~<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/.ssh/id</span>_rsa.pub  root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@hadoop03</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:~</span>
scp ~<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/.ssh/id</span>_rsa.pub  root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@hadoop04</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:~</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

scp 
从节点中执行如下命令,将主节点的公钥追加到授权文件中

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">cat  ~/id_rsa<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.pub</span> >>  ~/<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ssh</span>/authorized_keys
chmod <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">600</span> ~/<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ssh</span>/authorized_keys
rm ~/id_rsa<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.pub</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

从节点 
主节点中执行如下命令,实现主节点到从节点无密码访问

<code class="language-shell hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ssh hadoop02
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span>
ssh hadoop03
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span>
ssh hadoop04
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">exit</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

这里写图片描述

第5步:从节点到主节点无密码ssh访问

在每个从节点上执行如下命令,实现从节点到主节点的无密码访问

<code class="language-shell hljs dos has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cd</span>  
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cd</span>  .ssh
ssh-<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">copy</span>-id -i id_rsa.pub root@hadoop01</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

这里写图片描述

在每个从节点上执行如下命令,验证从节点到主节点无密码ssh访问。

<code class="language-shell hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ssh hadoop01</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

这里写图片描述

Logo

更多推荐