Test Procedure
The following uses 4000 files (each 150 MB) as an example. Run the following command on the server1 node to perform the test:
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.0-tests.jar TestDFSIO -write -nrFiles 4000 -size 150MB -erasureCodePolicy RS-6-3-1024k
- -write indicates a write test.
- -nrFiles 4000 indicates 4000 files.
- -size 150MB indicates that the size of each file is 150 MB.
- -erasureCodePolicy RS-6-3-1024k indicates that the RS-6-3-1024k EC policy is used.
Result example:
- Date & time indicates the test end time.
- Number of files indicates the number of test files.
- Total MBytes processed indicates the total volume of test data, in MB.
- Throughput mb/sec indicates the throughput, in MB/s. Calculation method: Total test data volume/Sum of the time each map task takes to write data
- Average IO rate mb/sec indicates the average I/O rate, in MB/s. Calculation method: [Sum of (Data volume each map task needs to write/The time each map task takes to write data)]/Number of tasks
- IO rate std deviation indicates the standard deviation of the I/O rate.
- Test exec time sec indicates the test execution duration, in seconds.
The Test exec time sec indicator is important. You can calculate the throughput of the entire test task as follows: Total volume of test data/Test execution time
Parent topic: KSAL EC Acceleration