Configuring HiBench

conf/hibench.conf

hibench.streambench.datagen.intervalSpan

50

50

Data generation interval.

hibench.streambench.datagen.recordsPerInterval

50000

50000

Number of data records generated each time.

hibench.streambench.datagen.recordLength

1000

1000

Size of generated data (byte).

hibench.streambench.kafka.topicPartitions

36

36

Number of partitions used for generating topics.

hibench.streambench.datagen.producerNumber

5

4

Number of Kafka producer threads.

hibench.streambench.kafka.home

/usr/hdp/current/kafka-broker

/usr/hdp/current/kafka-broker

Kafka home directory.

hibench.streambench.zkHost

ambari-server1:2181,ambari-agent2:2181,ambari-agent3:2181/kafka

server1:2181,agent2:2181,agent3:2181/kafka

Kafka ZooKeeper address.

hibench.streambench.kafka.brokerList

ambari-agent1:6667,ambari-agent2:6667,ambari-agent3:6667

agent1:6667,agent2:6667,agent3:6667

Kafka broker address.

conf/hadoop.conf

hibench.hadoop.home

/usr/hdp/current/hadoop-client

/usr/hdp/current/hadoop-client

Hadoop home directory.

hibench.hdfs.master

hdfs://ambari-server1:8020

hdfs://server1:8020

HDFS address.

conf/storm.conf

hibench.streambench.storm.nimbus

ambari-server1

server1

Nimbus IP address.

hibench.streambench.storm.home

/usr/hdp/current/storm-nimbus

/usr/hdp/current/storm-nimbus

Storm home directory.

hibench.streambench.storm.worker_count

4

4

Number of Storm workers.

hibench.streambench.storm.spout_threads

16

16

Number of spout threads.

hibench.streambench.storm.bolt_threads

24

24

Number of bolt threads.

conf/hibench.conf

hibench.streambench.datagen.intervalSpan

50

50

Data generation interval.

hibench.streambench.datagen.recordsPerInterval

50000

50000

Number of data records generated each time.

hibench.streambench.datagen.recordLength

1000

1000

Size of generated data (byte).

hibench.streambench.kafka.topicPartitions

36

36

Number of partitions used for generating topics.

hibench.streambench.datagen.producerNumber

5

3

Number of Kafka producer threads.

hibench.streambench.kafka.home

/usr/hdp/current/kafka-broker

/usr/hdp/current/kafka-broker

Kafka home directory.

hibench.streambench.zkHost

ambari-server1:2181,ambari-agent2:2181,ambari-agent3:2181/kafka

server1:2181,agent2:2181,agent3:2181/kafka

Kafka ZooKeeper address.

hibench.streambench.kafka.brokerList

ambari-agent1:6667,ambari-agent2:6667,ambari-agent3:6667

agent1:6667,agent2:6667,agent3:6667

Kafka broker address.

conf/hadoop.conf

hibench.hadoop.home

/usr/hdp/current/hadoop-client

/usr/hdp/current/hadoop-client

Hadoop home directory.

hibench.hdfs.master

hdfs://ambari-server:8020

hdfs://server1:8020

HDFS address.

conf/storm.conf

hibench.streambench.storm.nimbus

ambari-server

server1

Nimbus IP address.

hibench.streambench.storm.home

/usr/hdp/current/storm-nimbus

/usr/hdp/current/storm-nimbus

Storm home directory.

hibench.streambench.storm.worker_count

6

4

Number of Storm workers.

hibench.streambench.storm.spout_threads

12

16

Number of spout threads.

hibench.streambench.storm.bolt_threads

24

26

Number of bolt threads.

conf/hibench.conf

hibench.streambench.kafka.home

/usr/hdp/current/kafka-broker

Kafka installation directory.

hibench.streambench.zkHost

ambari-server1:2181,ambari-agent2:2181,ambari-agent3:2181/kafka

ZooKeeper address of Kafka. The value is in the format of Hostname:Port.

hibench.streambench.kafka.brokerList

ambari-agent1:6667,ambari-agent2:6667,ambari-agent3:6667

Addresses of all Kafka broker nodes. The value is in the format of hostname:port,hostname:port.

hibench.streambench.kafka.consumerGroup

HiBench

Group name of the Kafka consumer.

hibench.streambench.kafka.topicPartitions

30

Number of partitions used for generating topics.

hibench.streambench.kafka.offsetReset

largest

Start offset of the Kafka consumer.

hibench.streambench.datagen.intervalSpan

50

Interval span, in milliseconds.

hibench.streambench.datagen.recordsPerInterval

50000

Number of records generated for each interval span.

hibench.streambench.datagen.recordLength

1000

Fixed length of a record.

hibench.streambench.datagen.producerNumber

3

Number of Kafka producers running on different threads. The maximum rate of a single Kafka producer is about 100 Mbit/s.

hibench.streambench.datagen.totalRounds

-1

Total number of rounds for sending data.

hibench.streambench.datagen.totalRecords

-1

Total number of generated records.

hibench.streambench.datagen.dir

${hibench.hdfs.data.dir}/Streaming

Default path for storing seed files.

conf/storm.conf

hibench.streambench.storm.nimbus

server1

Name of the Nimbus host in the cluster.

hibench.streambench.storm.nimbusAPIPort

6627

Port of Storm Nimbus.

hibench.streambench.storm.home

/usr/hdp/current/storm-nimbus

Storm installation directory.

hibench.streambench.storm.worker_count

4

Number of Storm workers. The number of threads for most bolts is equal to the value of this parameter.

hibench.streambench.storm.spout_threads

16

Number of Kafka spout threads in Storm.

hibench.streambench.storm.bolt_threads

26

Total number of bolt threads.

hibench.streambench.storm.localshuffle

true

Indicates whether to perform shuffle directly on the node.

hibench.streambench.storm.nimbusContactInterval

10

Interval for determining whether the communication with Nimbus is complete.

conf/hadoop.conf

hibench.hadoop.home

/usr/hdp/current/hadoop-client

Hadoop installation path.

hibench.hdfs.master

hdfs://server1:8020

Address of the HDFS master.

hibench.hadoop.executable

${hibench.hadoop.home}/bin/hadoop

Path of the Hadoop execution file.

hibench.hadoop.configure.dir

${hibench.hadoop.home}/etc/hadoop

Path of the Hadoop configuration file.

hibench.hadoop.release

apache

Hadoop release version.