Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Configuring Kafka

  1. Configure log.dir.

    This parameter specifies the Kafka data storage directories (you can set multiple directories). New partitions are created in the directory with fewest partitions. Currently, 23 physical drives are allocated to Kafka. You need to set this parameter so that Kafka can use the drives. The configuration is as follows:

    1
    		 
    /srv/BigData/hadoop/data1/kafka-logs,/srv/BigData/hadoop/data2/kafka-logs,/srv/BigData/hadoop/data3/kafka-logs,/srv/BigData/hadoop/data4/kafka-logs,/srv/BigData/hadoop/data5/kafka-logs,/srv/BigData/hadoop/data6/kafka-logs,/srv/BigData/hadoop/data7/kafka-logs,/srv/BigData/hadoop/data8/kafka-logs,/srv/BigData/hadoop/data9/kafka-logs,/srv/BigData/hadoop/data10/kafka-logs,/srv/BigData/hadoop/data11/kafka-logs,/srv/BigData/hadoop/data12/kafka-logs

    The following figure shows the location of the parameter on the web page.

  2. Configure num.partitions.

    This parameter specifies the number of automatically created topic partitions. The default value is 2. It is tested that the performance is optimal when the value is 36.

    The following figure shows the location of the parameter on the web page.

  3. Configure num.network.threads.

    This parameter specifies the number of threads used by the broker to process network requests. The maximum value is 128. It is tested that the performance is optimal when the value is 128.

  4. Configure num.io.threads.

    Increase the number of threads for the broker to process drive I/Os. It is tested that the delay is optimal when the value is 8.

    The following figure shows the location of the parameter on the web page.

  5. Set JVM parameters.

    Add the following configuration to the /etc/profile file on all hosts to effectively reduce the number of JVM GCs:

    1
    		 
    export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"
Parent topic: Cases