方案实现

软件版本

项目

版本

Hadoop

apache hadoop-2.7.2

代码实现

实现该功能,需修改Hadoop源码中的FSDownload.java文件。

FSDownload.java原始代码可参考:https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java

  1. 修改“hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java”文件,新增aarch64PyMix(File dst)方法,该方法实现在ARM平台上将解压文件改名。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    private void aarch64PyMix(File dst) {  
      String dstPath = dst.getAbsolutePath();  
      String fileFullName = dst.getName();  
      String fileName = fileFullName.substring(0,fileFullName.length() - 4);  
      String orgDirName = dstPath + File.separator + fileName;  
      String aarch64DirName = dstPath + File.separator + fileName + "_aarch64";
      File orgFile = new File(orgDirName);
      File aarch64File = new File(aarch64DirName);
      if (orgFile.exists() && aarch64File.exists() && "aarch64".equals(System.getProperty("os.arch"))) {
         orgFile.renameTo(new File(orgDirName + "_bak"));
         aarch64File.renameTo(new File(orgDirName));
      }
    }
    

  2. “hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java”文件中,zip在unpack方法中的ARCHIVE条件下解压后,并增加对aarch64PyMix方法的调用。

    1
    2
    3
    4
    5
    6
    7
    8
    ...
    case ARCHIVE:
    ...  
      } else if (lowerDst.endsWith(".zip")) {
        FileUtil.unZip(localrsrc, dst);
        aarch64PyMix(dst);
      } else if (lowerDst.endsWith(".tar.gz")) ||
    ...