Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Compiling the Code

Obtaining the Adaptation Code Spark-ml-algo-lib of the Machine Learning Algorithm Library

Compiling the adaptation code for the machine learning algorithm library

The adaptation code is developed based on the open source software Spark 2.3.2, Spark 2.4.6, and Spark 3.1.1. It is used for compiling the machine learning algorithm library. Download the open source repository code that adapts to Spark 2.3.2, Spark 2.4.6, or Spark 3.1.1 to a specified directory, for example, /opt/, and decompress the package. (The following uses the package that adapts to Spark 2.3.2 as an example.) 
1
2
 
cd /opt/
unzip Spark-ml-algo-lib-v2.2.0-spark2.3.2.zip

The adaptation code of the machine learning algorithm library is built by incorporating some native code files of Spark 2.3.2, Breeze 0.13.1, netlib-2.2.1, and XGBoost 1.1.0 into the patch. For details about how to build the code, see Building Adaptation Code for the Machine Learning Algorithm Library.

Compiling the Open Source Adaptation Code for Algorithms Other Than XGBoost

  1. Go to the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ directory and compile the JAR packages required by the algorithms.
    1
2
    		 
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/
mvn clean package

    Before performing this step, ensure that the Internet connection is available. If the Internet is not accessible, an error will be reported when you run the command.

  2. Obtain the boostkit-ml-core_2.11-2.2.0-spark2.3.2.jar file from the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-core/target/ directory.
    1
    		 
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-core/target/

  3. Obtain the boostkit-ml-acc_2.11-2.2.0-spark2.3.2.jar file from the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-accelerator/target/ directory.
    1
    		 
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-accelerator/target/

  4. Obtain the boostkit-ml-kernel-client_2.11-2.2.0-spark2.3.2.jar file from the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-kernel-client/target/ directory.
    1
    		 
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-kernel-client/target/

Compiling the Open Source Adaptation Code for XGBoost

  1. To compile the code for XGBoost, go to the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages directory.
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages/
  2. Compile the JAR files used for the XGBoost algorithm.
    mvn package

  3. Obtain the boostkit-xgboost4j-spark2.3.2_2.11-2.2.0.jar file from the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages/boostkit-xgboost4j-spark/target/ directory.
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages/boostkit-xgboost4j-spark/target/

  4. Obtain the boostkit-xgboost4j_2.11-2.2.0.jar file from the /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages/boostkit-xgboost4j/target/ directory.
    cd /opt/Spark-ml-algo-lib-v2.2.0-spark2.3.2/ml-xgboost/jvm-packages/boostkit-xgboost4j/target/