11.1.7 Azkaban

azkaban双集群部署

在azkaban exec节点，以下服务的client均需部署两套

yarn(使用新实例client)

hive(使用新实例client)

spark(使用新实例client)

修改以下服务依赖为双依赖：

hdfs

yarn

hive

spark

对于旧版本（eayops<1.7.2.1）部署的azkaban需要重新load spec文件后，切换为新版本默认配置组

并增加以下依赖

* easy_alert

修改以下配置

global：

* enable_dev_cluster：true

以上操作处理完后重启并同步配置

在测试集群的hdfs上创建以下目录

- path: "/azkaban"
  owner: "mammut"
  group: "hadoop"
  mode: "0755"
- path: "/public"
  owner: "azkaban"
  group: "hadoop"
  mode: "1777"
- path: "/azkaban/resource"
  owner: "mammut"
  group: "hadoop"
  mode: "0755"
- path: "/public/sqoop_lib"
  owner: "azkaban"
  group: "hadoop"
  mode: "1777"

具体操作指令参考(注意替换路劲中的xxx和keytab对应的principal)

sudo -iu hdfs
   cd /home/bdms/hdfs/xxx/client/xxx/keytab
   kinit -kt hdfs.keytab hdfs/hadoop5422.jd.163.org
   cd ../config
   source hadoop-env.sh
   cd ../current/bin
   ./hdfs dfs -mkdir /azkaban
   ./hdfs dfs -chown mammut:hadoop /azkaban
   ./hdfs dfs -chmod 0755 /azkaban
   ./hdfs dfs -mkdir /azkaban/resource
   ./hdfs dfs -chown mammut:hadoop /azkaban/resource
   ./hdfs dfs -chmod 0755 /azkaban/resource
   ./hdfs dfs -mkdir /public
   ./hdfs dfs -chown azkaban:hadoop /public
   ./hdfs dfs -chmod 1777 /public
   ./hdfs dfs -mkdir /public/sqoop_lib
   ./hdfs dfs -chown azkaban:hadoop /public/sqoop_lib
   ./hdfs dfs -chmod 1777 /public/sqoop_lib

将以下路径的jar包上传到测试集群的hdfs的/public/sqoop_lib目录,（xxx根据实际情况替换）

/home/bdms/azkaban/xxx/exec/xxxx/package/hdfs_sqoop_lib-3.0.0-5.4， 可修改/home/bdms/azkaban/xxx/exec/xxxx/package/sqoop_lib/upload-sqoop-hdfs.sh后进行上传

HADDOP_HOME
HADOOP_CONF_DIR
修改为测试集群yarn client的current,config所在路径
cd /home/bdms/azkaban/xxx/exec/xxxx/package/sqoop_lib
以azkaban用户执行 ./upload-sqoop-hdfs.sh