Hadoop源码学习-编译源码
编译源码非常简单,步骤如下:下载源代码安装依赖软件或库编译打包下载源码官方下载并解压,过程略查看编译要求切换到解压后的hadoop源码根目录此目录下有个BUILDING.txt文件,里面有对building的详细说明,以Linux为例,要求如下:Unix SystemJDK 1.7+Maven 3.0 or laterFindbugs ...
编译源码非常简单,步骤如下:
- 下载源代码
- 安装依赖软件或库
- 编译打包
下载源码
官方下载并解压,过程略
查看编译要求
切换到解压后的hadoop源码根目录
此目录下有个BUILDING.txt
文件,里面有对building的详细说明,以Linux为例,要求如下:
- Unix System
- JDK 1.7+
- Maven 3.0 or later
- Findbugs 1.3.9 (if running findbugs)
- ProtocolBuffer 2.5.0
- CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
- Zlib devel (if compiling native code)
- openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
- Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
- Internet connection for first build (to fetch all Maven and Hadoop dependencies)
以上并不是每个都需要安装,看你编译的要求,我选择的在编译的时候,创建源码和二进制包,以及文档,我的环境如下:
* CentOS7
* JDK 1.8
* Maven 3.0.5
* Findbugs 1.3.9
* ProtocolBuffer 2.5.0
* CMake 3.6.3
* Zlib devel
* openssl devel
检查一下自己的系统,哪些安装过,哪些没有安装,安装之后需要可以直接在终端执行命令,所以,如果是那些网上手动下载解压的软件,需要对其配置PATH
变量。
以上软件或依赖,请自行全部安装,否则,在build的时候会报错,以下是部分软件安装过程:
安装Protoc
下载地址:
http://pan.baidu.com/s/1pJlZubT
解压,切换到根目录,安装命令如下:
./configure
make && make install
默认情况下,protoco是安装在/usr/local/bin',
/usr/local/man’下,但也可以指定安装到某个路径下面,例如,我要把protobuf安装在/home/yang/Soft/protobuf/
目录下,则使用配置项prefix
,如下:
./configure --prefix=/home/yang/Soft/protobuf/
make && make install
配置环境变量:
[yang@master bin]$ vim ~/.bashrc
export PATH=$PATH:/home/yang/Soft/protobuf/bin
[yang@master bin]$ source ~/.bashrc
[yang@master ~]$ protoc --version
libprotoc 2.5.0
安装Cmake(我之前安装过,只是没有配置PATH
,这里只是配置一下环境变量)
export PATH=$PATH:/home/yang/GitHub/json2avro/:/home/yang/Soft/protobuf/bin:/home/yang/Soft/cmake/cmake-3.6.3-Linux-x86_64/bin
安装findbugs
下载地址:https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/findbugs-1.3.9.tar.gz/download
这个也要配置环境变量
安装zlib
yum install zlib-devel.x86_64
安装openssl-devel
yum install openssl-devel.x86_64
编译打包
准备工作做完之后,切换到源码根目录,执行编译打包命令:
mvn package -Pdist,native,docs -DskipTests -Dtar
这个过程,时间会比较久,如果编译成功,有如下界面:
BUILD SUCCESS
Congratulations
Exception
java.io.IOException: Cannot run program “cmake”
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory
解决方法:
安装Cmake
hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn’t exist
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (site) on project hadoop-common: An Ant BuildException has occured: stylesheet /home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist.
[ERROR] around Ant part ...<xslt in="/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/findbugsXml.xml" style="${env.FINDBUGS_HOME}/src/xsl/default.xsl" out="/home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/site/findbugs.html"/>... @ 43:261 in /home/yang/GitHub/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
编辑.bashrc
文件,设置FINDBUGS_HOME
export FINDBUGS_HOME=/home/yang/Soft/findbugs-1.3.9
更多推荐
所有评论(0)