Tools

The HiBench test tool combines Storm with Kafka to reflect the end-to-end execution performance of Storm. Kafka functions as the data source. Storm uses the KafkaSpout class to consume data generated by Kafka and execute different topology tasks. The wordcount, identity, and fixwindow test cases are involved.

This test case focuses on the end-to-end performance of Storm. The involved configuration files are hadoop.conf, hibench.conf, and storm.conf. Figure 1 shows the test process.

The process is described as follows: Generate seed data in HDFS, use the dataGen script to submit the topic to Kafka, write the data to Kafka, submit the Storm topology tasks, start to consume data in Kafka, write the performance data back to Kafka, and run the metrics_read.sh script to read performance data from the topic.