修改配置

修改hadoop.conf

1. 打开配置文件。

   vi conf/hadoop.conf.template hadoop.conf

2. 按“i”进入编辑模式，根据实际情况对如下内容进行修改。

   # Hadoop home
       hibench.hadoop.home     /usr/hdp/current/hadoop-client
   # The root HDFS path to store HiBench data
       hibench.hdfs.master       hdfs://hadoop102:8020
    # Hadoop release provider. Supported value: apache, cdh5, hdp
        hibench.hadoop.release    hdp 

3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。

修改spark.conf

1. 打开配置文件。

   cp spark.conf.template spark.conf
   vi spark.conf

2. 按“i”进入编辑模式，根据实际情况对如下内容进行修改。

   # Spark home
   hibench.spark.home      /usr/hdp/current/spark2-client
   
   # executor number and cores when running on Yarn
   hibench.yarn.executor.num     20
   hibench.yarn.executor.cores   19
   
   # executor and driver memory in standalone & YARN mode
   spark.executor.memory  44g
   spark.driver.memory    36g

3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。

修改hibench.conf

1. 打开配置文件。

   1	vi hibench.conf

2. 按“i”进入编辑模式，根据实际情况对如下内容进行修改。

   # The definition of these profiles can be found in the workload's conf file i.e. conf/workloads/micro/wordcount.conf
   hibench.scale.profile                small #此处的small对应HiBench-HiBench-7.0/conf/workloads/micro/wordcount.conf里面设置的值
   # Mapper number in hadoop, partition number in Spark
   hibench.default.map.parallelism         8
   
   # Reducer nubmer in hadoop, shuffle partition number in Spark
   hibench.default.shuffle.parallelism     8
   
   #进入HiBench-HiBench-7.0/conf/workloads/micro/wordcount.conf修改对应级别的数据量

3. 按“Esc”键，输入:wq!，按“Enter”保存并退出编辑。

查看wordcount.conf目录

1

cat workloads/micro/wordcount.conf

#datagen
hibench.wordcount.tiny.datasize                 32000
hibench.wordcount.small.datasize                320000000
hibench.wordcount.large.datasize                3200000000
hibench.wordcount.huge.datasize                 32000000000
hibench.wordcount.gigantic.datasize             320000000000
hibench.wordcount.bigdata.datasize              1600000000000