Tuning Example
This example describes how to configure some parameters in a tuning scenario, including the Kafka server parameters and the client parameters.
Kafka Parameters
|
Component |
Parameter |
Recommended Value |
Description |
|---|---|---|---|
|
Kafka->Broker |
log.dir |
/hadoop/data1/kafka-logs,/hadoop/data2/kafka-logs,/hadoop/data3/kafka-logs ... /hadoop/data10/kafka-logs,/hadoop/data11/kafka-logs,/hadoop/data12/kafka-logs |
Set the number of disks used by Kafka to 12 to prevent disk read and write from becoming a bottleneck. |
|
Kafka->Broker |
num.network.threads |
128 |
Increase the number of Kafka network threads. |
|
Kafka->Broker |
num.io.threads |
128 |
Increase the number of Kafka I/O threads to improve the drive read and write capabilities. |
log.dir configuration path:
num.network.threads configuration path:
num.io.threads configuration path:
Tuning Tool Parameters
This section lists only the tuning parameters that need to be modified.
Kafka topic
1
|
replication-factor: 2 # Number of Kafka copies (2). |
Data production
1 2 |
Partitions: 36 # Number of data partitions. parallel: 6 # Number of producer processes on the client nodes (10 for x86). |
Data consumption
1 2 3 |
consumer_parallel: 12 # Number of consumer processes on the client nodes (12 for x86). fetch_threads: 10 # Number of threads for the consumer threads to obtain data (240 for x86). threads: 500 # Number of consumer threads. |
The configuration information is as follows:
