Tuning the TestDFSIO Tool

In HDFS cold storage, the drive bandwidth has reached the upper limit. The compression feature can be used to improve the bandwidth.

Purpose

Adjust the test file generation logic of the TestDFSIO tool provided by Hadoop. TestDFSIO generates test files of a specified size on local data nodes following certain rules. The files have a high compression ratio, which affects the compression test result. It needs to be adapted to achieve the following objectives:

Test files can be generated based on specified files as templates instead of specific rules.
Template files can be selected as required to flexibly adjust the compression ratio of test files.

Procedure

Go to the Hadoop source code directory.
```
cd ..\hadoop-3.1.0-src\
```
Obtain the patch.
Use a transfer tool to upload the patch to the ../hadoop-3.1.0-src/ directory.
Apply the patch.
```
patch -p1 < hdfs-testdfsio-0001.patch
```
Compile the Hadoop source code and deploy Hadoop. For details, see Compiling Hadoop and Deploying Hadoop.
Create a testFiles directory in the Hadoop running directory to store test files.
```
mkdir -p ../hadoop/testFiles
```

Copy test files to testFiles and create soft links (target_file is used as an example).

mv target_file ../hadoop/testFiles
cd ../hadoop/testFiles
ln -s target_file test_io_file

Repeat steps 5 to 7 on all servers.

Restart the Hadoop cluster.

/usr/local/hadoop/sbin/stop-all.sh
/usr/local/hadoop/sbin/start-all.sh
hdfs dfsadmin -safemode leave

Parent topic: Compression