Kylin4对接MRS_3.0.5非安全集群

MRS_3.0.5非安全集群对接Kylin4

一、准备资源

1.准备集群外ECS节点：

需要修改安全组，把节点网络和集群网络打通；

在节点上创建omm用户:  useradd -d /home/omm omm

2.准备MySQL环境；

创建MySQL数据库并和集群外ECS节点打通网络；

准备连接MySQL的java驱动包；

3.下载Kylin4安装包；

下载地址：https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.1/apache-kylin-4.0.1-bin-spark2.tar.gz

4.下载Spark 2.4.7安装包；

下载地址：https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz

5.下载依赖的jar包

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-beeline/1.2.1.spark_2.3.2-mrs-2.0/hive-beeline-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-cli/1.2.1.spark_2.3.2-mrs-2.0/hive-cli-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-common/1.2.1.spark_2.3.2-mrs-2.0/hive-common-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-exec/1.2.1.spark_2.3.2-mrs-2.0/hive-exec-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-jdbc/1.2.1.spark_2.3.2-mrs-2.0/hive-jdbc-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-metastore/1.2.1.spark_2.3.2-mrs-2.0/hive-metastore-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-shims/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-common/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-common-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-0.23/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-0.23-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/shims/hive-shims-scheduler/1.2.1.spark_2.3.2-mrs-2.0/hive-shims-scheduler-1.2.1.spark_2.3.2-mrs-2.0.jar

https://mirrors.huaweicloud.com/repository/maven/huaweicloudsdk/org/spark-project/hive/hive-serde/1.2.1.spark_2.3.2-mrs-2.0/hive-serde-1.2.1.spark_2.3.2-mrs-2.0.jar

https://dlcdn.apache.org//commons/configuration/binaries/commons-configuration-1.10-bin.zip

https://search.maven.org/remotecontent?filepath=org/apache/spark/spark-hive_2.11/2.4.7/spark-hive_2.11-2.4.7.jar

https://search.maven.org/remotecontent?filepath=org/apache/spark/spark-hive-thriftserver_2.11/2.4.7/spark-hive-thriftserver_2.11-2.4.7.jar

二、安装MRS客户端

在集群外节点安装MRS客户端，具体安装方法见MRS官网：https://support.huaweicloud.com/function-mrs/index.html

三、部署Kylin4对接MRS集群

1.上传附件kylin4_adapter.tar.gz到集群外节点，并把前面下载的依赖jar放到解压后的jars目录下；

2.解压并执行脚本进行适配

sh install.sh MRS客户端路径  kylin路径  spark开源解压包路径

例如：sh install.sh /opt/auto/mrsclient /opt/auto/apache-kylin-4.0.1-bin-spark2 /opt/auto/spark-2.4.7-bin-without-hadoop

四、配置Kylin4环境

参考上一步脚本执行完的提示信息进行配置；

五、使用Kylin4

1.执行如下命令启动kylin

source /xxx/xxx/bigdata_env

$KYLIN_HOME/bin/kylin.sh start

2.访问kylin页面地址

http://xxx.xxx.xxx.xxx:7070/kylin/jobs

六、注意事项：

1.本文是MRS 305版本非安全集群集成kylin的文档;

2.此文档默认是用omm作为执行用户，如果更改用户需要保证用户有Kylin配置的HDFS路径的权限；

附件： kylin4_adapter.rar 2.07KB 下载次数：0次

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。