文档
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Compiling Hadoop

  1. Download the Hadoop source package.
    1
    wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz
  2. Decompress the source package.
    1
    tar -zxvf hadoop-3.1.1-src.tar.gz
  3. Go to the directory generated after the decompression.
    1
    cd hadoop-3.1.1-src
  4. Modify the pom.xml file.
    1. Open the file.
      1
      vi pom.xml
    2. Press i to enter the insert mode and add the Kunpeng Maven repository source to the first line of the <repositories> tag.
      1
2
3
4
5
         <repository>
        <id>kunpengmaven</id>
        <name>kunpeng maven</name>
        <url>https://mirrors.huaweicloud.com/kunpeng/maven</url>
   </repository>
    3. Add the plugin repositories. <pluginRepositories> and <repositories> are at the same level.
      1
2
3
4
5
6
7
8
9
      <pluginRepositories>
  <pluginRepository>
    <id>huaweicloud-plugin</id>           
    <url>http://mirrors.huaweicloud.com/repository/maven</url>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </pluginRepository>
</pluginRepositories>
    4. Press Esc, type :wq!, and press Enter to save the file and exit.
  5. Optional: openEuler requires installing dependencies and applying patches.
    1. Install the dependencies.
      1
      yum -y install libtirpc libtirpc-devel
    2. Apply the patches.
      1
2
      wget https://issues.apache.org/jira/secure/attachment/12985484/HADOOP-15685.001.patch
patch -p1 < HADOOP-15685.001.patch
  6. Run the compile command.
    1
    mvn package -DskipTests -Pdist,native -Dtar -Dmaven.javadoc.skip=true
  7. Optional: Add the Snappy repository compile command.
    1
    mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true

    If you have followed Compiling Dependencies to install the Snappy dependency, run the compilation commands above.

    After the compilation is successful, the hadoop-3.1.1.tar.gz package is generated in the hadoop-dist/target/ directory of the source code.

  8. Use the Kunpeng Porting Advisor to scan the resulting TAR package, to ensure that no x86-specific .so files or JAR packages remain. For details, see Using the Kunpeng Porting Advisor.

    Use the Kunpeng Porting Advisor to scan the resulting TAR package, to verify that no x86-specific .so files or JAR packages remain. Otherwise, the x86-specific .so files or JAR files may cause functionality problems in subsequent operations.