Parameter Configuration
This example lists some parameter configurations in a Flink tuning scenario for reference only.
Web Configuration
|
Component |
Parameter |
Recommended Value |
Description |
|---|---|---|---|
|
Kafka->Broker |
log.dir |
/hadoop/data1/kafka-logs, /hadoop/data2/kafka-logs, /hadoop/data3/kafka-logs, ... /hadoop/data11/kafka-logs, /hadoop/data12/kafka-logs |
Set the number of disks used by Kafka to 12 to prevent disk read and write from becoming a bottleneck. |
|
Kafka->Broker |
num.partitions |
36 |
Use more partitions and the log.dir parameter to improve the Kafka performance, and the concurrency of the Flink data generator. |
|
Kafka->Broker |
num.network.threads |
128 |
Increase the number of Kafka network threads. |
|
Kafka->Broker |
num.io.threads |
8 |
The Flink data load is not large and the number of I/O threads is small to reduce the CPU pressure. |
|
Yarn->Nodemanager |
yarn.nodemanager.resource.memory-mb |
224256 |
Increase the maximum memory that can be allocated to each NodeManager of Yarn. |
|
Yarn->Nodemanager |
yarn.nodemanager.resource. cpu-vcores |
48 |
Increase the maximum number of vCores that can be allocated to each NodeManager of Yarn. |
|
Yarn->Nodemanager |
yarn.nodemanager.numa-awareness.enabled |
true |
Enable Yarn NUMA core binding. |
|
Yarn->Nodemanager |
yarn.nodemanager.numa-awareness.read-topology |
true |
Specifies whether to bind cores using the parameter or the startup command. For Flink, bind cores using the startup command. |
|
Yarn->Nodemanager |
yarn.nodemanager.numa-awareness.numactl.cmd |
/usr/bin/numactl |
System path of the NUMA core binding tool. |
Kafka parameter configuration path:
Yarn parameter configuration path:
Configuring Client
Only the optimization parameters that need to be modified are listed.
The Flink task submission configuration is as follows:
1 2 3 4 |
yarn-session.sh -s 16 -jm 5000 -tm 30000 -d slot: 7 (16 for x86) jobmanager.memory: 5000 taskmnager.memory: 30000 |
The yahoo_benchmark_streaming case configuration (by modifying stream-bench-hdp.sh) is as follows:
1 2 3 |
Partitions: 36 Load: 8600000 (6400000 for x86) TEST_TIME: 240 |