获取图分析算法加速库开发依赖JAR包

- 构建图分析算法加速库适配代码Spark-graph-algo-lib过程如下。此过程以适配Spark 2.3.2代码的构建为例,适配Spark 2.4.6/Spark 3.1.1代码与之相似,可参考以下操作。
- 以下操作请在Linux环境下操作,该章节仅供参考。
构建适配代码
图分析算法加速库开源仓的代码是基于Spark 2.3.2和图算法开源patch组成,构建过程:
- 下载Spark 2.3.2源码zip包到“/opt/”目录并解压,得到Spark源码目录“/opt/spark-2.3.2”。
1 2
wget https://github.com/apache/spark/archive/v2.3.2.zip unzip spark-2.3.2.zip
- 在“/opt/”目录下建立工程Spark-graph-algo-lib及相关目录。
1 2 3
cd /opt/ mkdir -p Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx mkdir -p Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature
- 按照表1的对应关系将Spark 2.3.2中的对应原文件复制到Spark-graph-algo-lib目录,表格左边两列是目标目录和文件名,右边两列的是需要移动的原文件目录及文件名,示例如下。操作命令示例:
1 2 3 4
cp /opt/spark-2.3.2/graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala /opt/ Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx cp /opt/spark-2.3.2/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala /opt/ Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature
表1 Spark中需要放入Spark-graph-algo-lib工程的文件 Spark-graph-algo-lib工程目录
Spark-graph-algo-lib工程文件名
Spark原文件所在目录
Spark原文件名
Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx
GraphOps.scala
spark-2.3.2/graphx/src/main/scala/org/apache/spark/graphx
GraphOps.scala
Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature
Word2Vec.scala
spark-2.3.2/mllib/src/main/scala/org/apache/spark/mllib/feature
Word2Vec.scala
- 下载并解压patch到“/opt/”目录下,得到完整的图分析算法加速库适配代码Spark-graph-algo-lib。
1 2 3 4 5
cd /opt/ wget https://github.com/kunpengcompute/Spark-graph-algo-lib/releases/download/v2.2.0-spark2.3.2/Spark-graph-algo-lib-Spark2.3.2.patch mv Spark-graph-algo-lib-Spark2.3.2.patch Spark-graph-algo-lib cd Spark-graph-algo-lib patch -p1 < Spark-graph-algo-lib-Spark2.3.2.patch
父主题: 参考