Kafka Parameter Configuration
Purpose
Kafka is an I/O component. You need to adjust the number of threads for the brokers to process messages and disk I/O. You should determine whether to use the compression algorithm and which compression algorithm to use based on the scenario requirements. In addition, you should set the number of partitions of Kafka based on the number of disks.
Procedure
Search for the following parameters on the Kafka web page and modify them:
Parameter |
Recommended Value |
Description |
|---|---|---|
num.network.threads |
128 |
This parameter specifies the maximum number of threads for the brokers to process messages. The brokers mainly process network I/Os and read and write buffer data. Currently, the bottleneck of Kafka is mainly the network. Therefore, this parameter has a great impact. The default value of this parameter is 3, which has a decisive impact on the Kafka performance. If this parameter is set to a larger value, the performance will be greatly improved. The recommended value of this parameter is the number of cores plus 1. However, due to the NIC bottleneck, the performance is not affected by this parameter when it is big enough. To facilitate debugging, you can set this parameter to the number of cores plus 1 or even to the maximum value (128). |
num.io.threads |
65 |
This parameter specifies the number of threads used by the brokers to process disk I/Os. Currently, the bottleneck of Kafka is mainly the network. Therefore, the number of threads used by the brokers to process disk I/Os does not have a great impact. In typical scenarios, 23 Kafka disks (SAS disks) are deployed for a single broker. The single-disk performance is high. Actually, three to four disks can cause the network I/Os to reach the bottleneck. Therefore, the modification of the number of I/O threads has limited impact on the performance. The default value of this parameter is 8, and the maximum value is 256. |
compression.type |
Adjusted the value based on the actual scenarios. |
This parameter specifies the Kafka compression algorithm selection. The value can be producer (determined by the client producer), uncompressed (not compressed), gzip, snappy, or lz4. According to the test results, the performance of the LZ4 and Snappy compression algorithms is good. The Gzip compression algorithm is not recommended. However, in most scenarios, "uncompressed" is better. The compression algorithm is sometimes a double-edged sword, which may not help. Therefore, the compression algorithm needs to be comprehensively considered in system scenarios. |
partitions |
Adjust the value based on the number of disks. Generally, the value is 1 to 2 times of the number of disks. |
This parameter specifies the number of partitions of Kafka topics. This parameter is closely related to topics. That is, the number of partitions for storing Kafka data must be greater than the total number of disks to ensure that each disk has read and write operations. However, there is no disk I/O bottleneck in actual services. Therefore, adjust the parameters based on your requirements. |