---
title: 构建机器学习算法加速库适配代码
description: "- 构建机器学习算法加速库适配代码Spark-ml-algo-lib过程如下。此过程以适配Spark 3.3.1代码的构建为例。"
url: https://www.hikunpeng.com/document/detail/zh/kunpengbds/appAccelFeatures/algorithmaccelf_ml/kunpengbdssparkml_20_0040.html
sourcePath: /source/zh/kunpengbds/appAccelFeatures/algorithmaccelf_ml/kunpengbdssparkml_20_0040.html
indexId: 259da55076a3309a7e62f88e578c40b7a422d1c90c3ecea7f8d47f987df7b81a88
---
# 构建机器学习算法加速库适配代码

- 构建机器学习算法加速库适配代码Spark-ml-algo-lib过程如下。此过程以适配Spark 3.3.1代码的构建为例。
- 以下操作请在Linux环境下操作，该章节仅供参考。


1. 下载Spark 3.1.1源码包(https://github.com/apache/spark/archive/v3.3.1.zip)到“/opt/”目录并解压，得到Spark源码目录。
  1 wget https://github.com/apache/spark/archive/v3.3.1.zip

2. 下载Breeze 0.13.1源码包(https://github.com/apache/spark/archive/v3.3.1.zip)到“/opt/”目录并解压，得到Breeze源码目录。
  1 wget https://github.com/scalanlp/breeze/archive/releases/v1.0.zip

3. 在“/opt/”目录下建立一个层级为如下所示的目录的工程Spark-ml-algo-lib。
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 cd /opt/ mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/classification mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/feature mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/fpm mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/recommendation mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/regression mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/tree/impl mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/tuning mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/clustering mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/feature mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/fpm mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/linalg/distributed mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/optimization mkdir -p Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/mllib/tree mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/breeze/numerics mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/ml/classification mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/ml/recommendation mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/ml/regression mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/ml/tree/impl mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/ml/tuning mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/mllib/clustering mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/mllib/feature mkdir -p Spark-ml-algo-lib/ml-core/src/main/scala/org/apache/spark/mllib/fpm

4. 按照表1、表2的对应关系将Spark 3.3.1、Breeze 1.0中的对应原文件复制到“Spark-ml-algo-lib”目录，表格左边两列是目标目录和文件名，右边两列的是需要移动的原文件目录及文件名。由于需要复制的文件很多，操作的代码只给出两个示例。
  有些文件在复制到目标文件夹后需要改名。 操作命令示例： 1 2 cp /opt/spark-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala /opt/Spark-ml-algo-lib/ml-accelerator/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala cp /opt/breeze-releases-v1.0/math/src/main/scala/breeze/numerics/package.scala /opt/Spark-ml-algo-lib/ml-core/src/main/scala/breeze/numerics/DigammaX.scala

**表1 Spark中需要放入Spark-ml-algo-lib工程的文件**

| Spark\-ml\-algo\-lib工程目录 | Spark\-ml\-algo\-lib工程文件名 | Spark原文件所在目录 | Spark原文件名 |
| --- | --- | --- | --- |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/classification/ | GBTClassifier.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/ | GBTClassifier.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/classification/ | LinearSVC.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/ | LinearSVC.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/classification/ | RandomForestClassifier.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/ | RandomForestClassifier.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/classification/ | DecisionTreeClassifier.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/ | DecisionTreeClassifier.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/classification/ | FMClassifier.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification/ | FMClassifier.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/feature | IDF.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/feature | IDF.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/feature | Word2Vec.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/feature | Word2Vec.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/feature | DecisionTreeBucketizer.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/classification | RandomForestClassifier.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/fpm | PrefixSpan.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/fpm | PrefixSpan.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/recommendation/ | ALS.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/recommendation | ALS.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/recommendation/ | NMF.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/recommendation | ALS.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/regression/ | DecisionTreeRegressor.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/regression/ | DecisionTreeRegressor.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/regression/ | GBTRegressor.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/regression/ | GBTRegressor.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/regression/ | FMRegressor.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/regression/ | FMRegressor.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/regression/ | RandomForestRegressor.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/regression/ | RandomForestRegressor.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | GradientBoostedTrees.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | GradientBoostedTrees.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest4GBDTX.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForestRaw.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | DecisionForest.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | DecisionTreeBucket.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | RandomForest.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/impl/ | DecisionTreeMetadata.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | DecisionTreeMetadata.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/ | treeParams.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/ | treeParams.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tree/ | treeModels.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/ | treeModels.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/ src/main/scala/org/apache/spark/ml/tuning/ | BayesianCrossValidator.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tuning/ | CrossValidator.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/clustering/ | LDA.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/clustering | LDA.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/clustering/ | LDAOptimizer.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/clustering | LDAOptimizer.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/feature | IDF.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature/ | IDF.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/feature | Word2Vec.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature/ | Word2Vec.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/feature | PCA.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature/ | PCA.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/fpm/ | PrefixSpan.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/fpm | PrefixSpan.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/fpm/ | FPGrowth.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/fpm | FPGrowth.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/linalg/distributed/ | RowMatrix.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed | RowMatrix.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/linalg/ | EigenValueDecomposition.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/linalg | EigenValueDecomposition.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/optimization/ | LBFGSN.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/optimization | LBFGS.scala |
| Spark\-ml\-algo\-lib/ml\-accelerator/src/main/scala/org/apache/spark/mllib/tree/ | DecisionTree.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/tree | DecisionTree.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/ | Node.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/ | Node.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/impl | BaggedPoint.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | BaggedPoint.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/impl | DTFeatureStatsAggregator.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | DTStatsAggregator.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/impl | GradientBoostedTreesCore.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | GradientBoostedTrees.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/impl | TreePointX.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | TreePoint.scala |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/org/apache/spark/ml/tree/impl | TreePointY.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/ml/tree/impl/ | TreePoint.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/clustering/ | LDAUtilsX.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/clustering | LDAUtils.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/clustering/ | OnlineLDAOptimizerXObj.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/clustering | LDAOptimizer.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/feature/ | VocabWord.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature | Word2Vec.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/fpm/ | LocalPrefixSpan.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/fpm/ | LocalPrefixSpan.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/fpm/ | PrefixSpanBase.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/fpm/ | PrefixSpan.scala |
| Spark\-ml\-algo\-lib/ml\-core/src/main/scala/org/apache/spark/mllib/fpm/ | FPGrowthCore.scala | spark\-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/fpm/ | FPGrowth.scala |


**表2 Breeze中需要放入Spark-ml-algo-lib工程的文件**

| Spark\-ml\-algo\-lib工程目录 | Spark\-ml\-algo\-lib工程文件名 | Breeze原文件所在目录 | Breeze原文件名 |
| --- | --- | --- | --- |
| Spark\-ml\-algo\-lib/ml\-core/ src/main/scala/breeze/numerics/ | DigammaX.scala | breeze\-releases\-v1.0/math/src/main/scala/breeze/numerics/ | package.scala |


5. 下载Spark-ml-algo-lib-3.3.1.patch(https://gitee.com/kunpengcompute/Spark-ml-algo-lib/releases/download/v3.0.0-Spark3.3.1/Spark-ml-algo-lib-3.3.1.patch)到“/opt/Spark-ml-algo-lib/”目录下，以Spark 3.3.1为例，将Spark 3.3.1的patch并入Spark-ml-algo-lib，得到完整的机器学习算法加速库适配代码Spark-ml-algo-lib。
  1 2 3 cd /opt/Spark-ml-algo-lib wget https://github.com/kunpengcompute/Spark-ml-algo-lib/releases/download/v3.0.0-spark3.3.1/Spark-ml-algo-lib-Spark3.3.1.patch patch -p1 < Spark-ml-algo-lib-Spark3.3.1.patch

  完整的机器学习算法加速库适配代码Spark-ml-algo-lib的目录与仓库代码一致。