文档
我要评分
获取效率
正确性
完整性
易理解
提交关闭

Compiling HiBench

  1. Download the HiBench source code.
    1
    		 
    wget https://github.com/Intel-bigdata/HiBench/archive/HiBench-7.0.tar.gz
  2. Decompress the source package.
    1
    		 
    tar -zxf HiBench-7.0.tar.gz -C ./
  3. Modify the autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIO.java file.
    1. Go to the HiBench-HiBench-7.0 directory and open the TestDFSIO.java file. 
      1
2
      		 
      cd HiBench-HiBench-7.0
vi autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIO.java
    2. Press i to enter the insert mode and replace import org.apache.commons.logging.*; with the following content: 
      import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
    3. Replace private static final Log LOG = FileInputFormat.LOG; with the following content: 
      private static final Logger LOG = LoggerFactory.getLogger(FileInputFormat.class);
    4. Press Esc, type :wq!, and press Enter to save the file and exit.
  4. Modify the autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIOEnh.java file.
    1. Open the TestDFSIOEnh.java file. 
      1
      		 
      vi autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIOEnh.java
    2. Press i to enter the insert mode and add the following content to the file header: 
      1
      		 
      import java.util.Arrays;
    3. Replace import org.apache.commons.logging.*; with the following content: 
      1
2
      		 
      import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
    4. Replace private static final Log LOG = LogFactory.getLog(TestDFSIOEnh.class); with the following content: 
      1
      		 
      private static final Logger LOG = LoggerFactory.getLogger(TestDFSIOEnh.class);
    5. Replace FileUtil.copyMerge(fs, DfsioeConfig.getInstance().getReportDir(fsConfig), fs, DfsioeConfig.getInstance().getReportTmp(fsConfig), false, fsConfig, null); with the following content: 
      1
      		 
      copyMerge(fs, DfsioeConfig.getInstance().getReportDir(fsConfig), fs, DfsioeConfig.getInstance().getReportTmp(fsConfig), false, fsConfig, null);
    6. Add two new methods before the analyzeResult method, as shown in the following: 
       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
      		 
          @Deprecated
       public static boolean copyMerge(FileSystem srcFS, Path srcDir, FileSystem dstFS, Path dstFile, boolean deleteSource,
                       Configuration conf, String addString) throws IOException {
        dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);
        if (!srcFS.getFileStatus(srcDir).isDirectory())
            return false;
               OutputStream out = dstFS.create(dstFile);
               try {
                       FileStatus contents[] = srcFS.listStatus(srcDir);
                       Arrays.sort(contents);
                       for (int i = 0; i < contents.length; i++) {
                               if (contents[i].isFile()) {
                                       InputStream in = srcFS.open(contents[i].getPath());
                                       try {
                                               IOUtils.copyBytes(in, out, conf, false);
                                               if (addString != null)
                                                       out.write(addString.getBytes("UTF-8"));
                                       } finally {
                                               in.close();
                                       }
                               }
                       }
               } finally {
                       out.close();
               }
               if (deleteSource) {
                       return srcFS.delete(srcDir, true);
               } else {
                       return true;
               }
    }
       private static Path checkDest(String srcName, FileSystem dstFS, Path dst, boolean overwrite) throws IOException {
               if (dstFS.exists(dst)) {
                       FileStatus sdst = dstFS.getFileStatus(dst);
                       if (sdst.isDirectory()) {
                               if (null == srcName) {
                                       throw new IOException("Target " + dst + " is a directory");
                               }
                               return checkDest(null, dstFS, new Path(dst, srcName), overwrite);
                       } else if (!overwrite) {
                               throw new IOException("Target " + dst + " already exists");
                       }
               }
               return dst;
       }
    7. Press Esc, type :wq!, and press Enter to save the file and exit.
  5. Modify the pom.xml file.
    1
    		 
    vi hadoopbench/mahout/pom.xml
  6. Press i to enter the insert mode and comment out the following code:

  7. Press Esc, type :wq!, and press Enter to save the file and exit.
  8. Perform the compilation.
    1
    		 
    mvn -Dspark=2.2 -Dscala=2.11 -Dhadoop.mr2.version=3.1.0 clean package