Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

HDFS Tuning

Perform the following operations on agent1 to agent9.

Optimizing JVM Compilation Parameters

  1. Change the number of JVM compilation threads to 2.
    1. Open mapred-site.xml.
      vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
    2. Press i to enter the insert mode and add the following content:
       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
      <property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx5530m -XX:CICompilerCount=2</value>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx2765m -XX:CICompilerCount=2</value>
</property>
<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx2048m -Xms2048m -XX:CICompilerCount=2</value>
</property>

      The default HDFS configuration contains mapreduce.map.java.opts, mapreduce.reduce.java.opts, and mapred.child.java.opts. You only need to add -XX:CICompilerCount=2 to the end of each value.

    3. Press Esc to exit the insert mode. Input :wq! and press Enter to save the file and exit.
  2. Restart HDFS.
    Run the following commands on the server1 node:
    /usr/local/hadoop/sbin/stop-all.sh
/usr/local/hadoop/sbin/start-all.sh
hdfs dfsadmin -safemode leave

Optimizing the BiSheng JDK

  1. Install the BiSheng JDK according to BiSheng JDK 8 Installation Guide.
  2. On the server1 node, change the JDK used by HDFS to BiSheng JDK.
    1. Open hadoop-env.sh.
      vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh
    2. Press i to enter the insert mode and modify the JAVA_HOME path.
      1
      export JAVA_HOME=/usr/local/bisheng-jdk1.8.0_422

      /usr/local/bisheng-jdk1.8.0_422 indicates the BiSheng JDK installation path. Change it to the actual one.

    3. Press Esc to exit the insert mode. Input :wq! and press Enter to save the file and exit.
  3. Copy hadoop-env.sh to agent nodes.
    1
2
3
4
5
6
7
8
9
    scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent1:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent2:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent3:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent4:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent5:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent6:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent7:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent8:/usr/local/hadoop/etc/hadoop/
scp /usr/local/hadoop/etc/hadoop/hadoop-env.sh agent9:/usr/local/hadoop/etc/hadoop/
  4. Restart HDFS.
    Run the following commands on the server1 node:
    /usr/local/hadoop/sbin/stop-all.sh
/usr/local/hadoop/sbin/start-all.sh
hdfs dfsadmin -safemode leave
Parent topic: HDFS Tuning Guide