Modifying Yarn/Kafka Parameters

yarn.nodemanager.resource.cpu-vcores

Number of cores on each Kafka service node

This parameter is available in yarn > CONFIGS > SETTINGS - Number of virtual cores on the web page. This parameter indicates the number of cpu-vcores that can be used by a NodeManager. Set this parameter to the number of cores of a node.

yarn.nodemanager.numa-awareness.enabled

true

By default, this parameter does not exist in the Yarn configuration. You need to manually add it in Yarn > CONFIGS - ADVANCED > Nodemanager > Custom yarn-site > Add Property on the web page. Indicates whether to enable the NUMA awareness function of a container. The default value is false. To enable the function, set this parameter to true.

yarn.nodemanager.numa-awareness.read-topology

true

By default, this parameter does not exist in the Yarn configuration. You need to manually add it in Yarn > CONFIGS - ADVANCED > Nodemanager > Custom yarn-site > Add Property on the web page. This parameter indicates whether to read the topology structure of NUMA from the system or configuration. If this parameter is set to true, the numactl -hardware command is used to read the NUMA structure of the system. Set this parameter to true if NUMA core binding needs to be enabled for Flink tests.

kafka.num.network.threads

8

This parameter is set in - Kafka > CONFIGS > Advanced kafka > broker > num.network.threads on the web page. This parameter indicates the number of Kafka network threads, which affects the Kafka read and write efficiency. However, Flink processing resources are limited. You are advised not to set this parameter to a large value. Otherwise, Kafka occupies too many resources, affecting Flink optimization performance. You can set this parameter to 1 to 2 times of the default value based on the site requirements.