Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Compiling Spark

Compile Spark on the Kunpeng platform and scan the generated TAR package to ensure that no legacy x86 files exist, to prevent potential functional issues.

The following uses Spark 2.3.0 as an example to describe how to compile Spark on the Kunpeng platform. These steps are applicable to other versions of Spark. You need to change the version number in the commands accordingly.

  1. Download the Spark source archive.
    1
    wget https://github.com/apache/spark/archive/v2.3.0.tar.gz
  2. Extract the archive and go to the resulting source code directory.
    1
2
    tar -zxf v2.3.0.tar.gz
cd spark-2.3.0
  3. Configure the Maven repository.
    1. Open the pom.xml file.
      1
      vi pom.xml
    2. Press i to enter the insert mode and add the Kunpeng Maven repository at the beginning of the <repositories> tag.
      1
2
3
4
5
      <repository>
    <id>kunpengmaven</id>
    <name>kunpeng maven</name>
    <url>https://mirrors.huaweicloud.com/kunpeng/maven</url>
</repository>
    3. Add the Huawei Maven repository at the beginning of the <pluginRepository> tag.
      1
2
3
4
      <pluginRepository>
    <id>huaweicloud-plugin</id>
    <url>https://mirrors.huaweicloud.com/repository/maven</url>
</pluginRepository>
    4. Press Esc, type :wq!, and press Enter to save the file and exit.
  4. Execute compilation.
    1
    dev/make-distribution.sh --tgz -Pyarn,hive,hive-thriftserver,spark

    After the compilation is complete, the compiled package spark-2.3.0-bin-2.6.5.tgz is generated in the source code directory.

    If the message Cannot find 'R_HOME' is displayed during the compilation, resolve the issue as instructed in Error "Cannot find'R_HOME" Reported When Compiling Spark.

  5. Use the Kunpeng Porting Advisor to scan the resulting TAR package, to ensure that no x86-specific .so files or JAR packages remain. For details, see Using the Kunpeng Porting Advisor.

    Use the Kunpeng Porting Advisor to scan the resulting TAR package, to verify that no x86-specific .so files or JAR packages remain. Otherwise, the x86-specific .so files or JAR files may cause functionality problems in subsequent operations.