Azkaban 安装部署

编译

从源代码构建
Azkaban构建使用Gradle（使用Gradle包装器gradlew运行时自动下载），并且需要Java 8或更高版本，还需要有git。

下面的命令上运行*如Linux，OS X. nix平台

#Install Git
#yum -y install git

# Build Azkaban
./gradlew build

# Clean the build
./gradlew clean

# Build and install distributions
./gradlew installDist

# Run tests
./gradlew test

# Build without running tests
./gradlew build -x test

这些都是标准的Gradle命令。请查看Gradle文档以获取更多信息。

Gradle在项目目录中创建.tar.gz文件。例如。./azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz。
官网说明

问题

./gradlew build

提示：

Could not determine the dependencies of task ':az-flow-trigger-dependency-type:kafka-event-trigger:fatJar'.
> Could not resolve all files for configuration ':az-flow-trigger-dependency-type:kafka-event-trigger:compile'
   > Could not download kafka_2.11.jar (org.apache.kafka:kafka_2.11:0.10.2.0)
      > Could not get resource 'https://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.11/0.10.2.0/kafka_2.11-0.10.2.0.jar'

后面省略n个错误

解决：我们需要在/az-flow-trigger-dependency-type/kafka-event-trigger目录运行。

# Build Azkaban
../../gradlew build

# Clean the build
../../gradlew clean

# Build without running tests
../../gradlew build -x test

官网说明
其他问题

安装

将编译好的的放在自己建的yum源

#将编译好的包复制到yum源目录
cp azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz /var/www/html/Package/
cp azkaban-web-server-0.1.0-SNAPSHOT.tar.gz /var/www/html/Package/
tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz 
cp create-all-sql-0.1.0-SNAPSHOT.sql /var/www/html/Package/
cd /var/www/html/Package/
#createrepo
createrepo .

构建ambari server

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
sudo git clone https://github.com/jzyhappy/ambari-azkaban-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/AZKABAN

重启ambari

ambari-server restart

创建mysql

1. 为Azkaban创建数据库：

# Example database creation command, although the db name doesn't need to be 'azkaban'
mysql> CREATE DATABASE azkaban;

2. 为Azkaban创建一个mysql用户。例如，：

# Example database creation command. The user name doesn't need to be 'azkaban'
mysql> CREATE USER 'username'@'%' IDENTIFIED BY 'password';
# give the user INSERT, SELECT, UPDATE, DELETE permission on all tables in the Azkaban db.
mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to '<username>'@'%' WITH GRANT OPTION;

3. Mysql数据包大小可能需要重新配置。默认情况下，MySQL的可允许数据包大小可能低得离谱。要增加它，您需要将属性max_allowed_pa​​cket设置为一个更大的数字，例如1024M。要在linux中配置它，请打开/etc/my.cnf。在mysqld之后的某处，添加以下内容：

[mysqld]
...
max_allowed_packet=1024M

4. 要重启MySQL，您可以运行：

$/bin/systemctl restart mysqld.service

修改配置

文件路径/configuration/

azkaban-db.xml

<property>
    <name>mysql.host</name>
    <value><your mysql server></value>
</property>

azkaban-executor.properties.xml

...
<property>
    <name>azkaban.webserver.url</name>
    <value>http://<your webserver>:10200</value>
</property>
...
<property>
    <name>mysql.host</name>
    <value><your mysql server></value>
</property>
...
#这步不用改，这里是解决我下面所说的问题添加的
<property>
    <name>executor.port</name>
    <value>12321</value>
</property>

azkaban-web.properties.xmll

<property>
    <name>mysql.host</name>
    <value><your mysql server></value>
</property>

修改文件

文件位置/package/scripts/
download.ini

azkaban_web_url = <your repo>/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz
azkaban_executor_url = <your repo>/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
azkaban_db_url = <you repo>/create-all-sql-0.1.0-SNAPSHOT.sql

问题

Azkaban Web Server启动一段时后自动停，查看日志

cat webServerLog_2020-02-06+15\:24\:03.out 
ERROR [ExecutorManager] No active executors found
ERROR [StdOutErrRedirect] Exception in thread "main" 
ERROR [StdOutErrRedirect] azkaban.executor.ExecutorManagerException: No active executors found
ERROR [StdOutErrRedirect] 	at azkaban.executor.ActiveExecutors.setupExecutors(ActiveExecutors.java:52)
ERROR [StdOutErrRedirect] 	at azkaban.executor.ExecutorManager.setupExecutors(ExecutorManager.java:201)
ERROR [StdOutErrRedirect] 	at azkaban.executor.ExecutorManager.initialize(ExecutorManager.java:135)
ERROR [StdOutErrRedirect] 	at azkaban.executor.ExecutorManager.start(ExecutorManager.java:149)
ERROR [StdOutErrRedirect] 	at azkaban.webapp.AzkabanWebServer.launch(AzkabanWebServer.java:231)
ERROR [StdOutErrRedirect] 	at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:224)

Azkaban Executor Server 启动是报错，但是在ambari上显示正常

看azkaban_executor.py line52 executor.prot 没有在配置文件中加上即可下面用的是手动的方式改的，这个前边会改，下载我的github不会出现问题了

查看数据库

mysql> use azkaban;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from executors;
+----+--------------+-------+--------+
| id | host         | port  | active |
+----+--------------+-------+--------+
|  2 | landrover-28 | 37842 |      0 |
+----+--------------+-------+--------+
1 row in set (0.00 sec)

active = 0 执行命令

curl http://landrover-28:37842/executor?action=activate

查看数据库

mysql> use azkaban;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from executors;
+----+--------------+-------+--------+
| id | host         | port  | active |
+----+--------------+-------+--------+
|  3 | landrover-28 | 37842 |      1 |
+----+--------------+-------+--------+
1 row in set (0.00 sec)

重启web server 成功