Calculating Executor Parameters
Purpose
The goal of calculating heuristic executor parameters is to maximize CPU and memory utilization, thereby improving overall task execution efficiency.
Procedure
The actual testing shows that Kunpeng's NUMA architecture and Yarn's NUMA awareness significantly impact performance. To achieve balanced resource utilization, containers (executors) should be evenly distributed across NUMA nodes. Furthermore, the total number of executors should be determined as a multiple of the compute node count.
- Determine the NUMA topology of the Kunpeng processor (for example, 2 NUMA nodes) and the number of Yarn compute nodes (for example, 3 compute nodes) to derive the base multiplier for the total executor count. For instance, in a setup with 2 NUMA nodes and 3 compute nodes, the number of executors should be a multiple of 6.
- Determine the executor count range.
Multiply the number of NUMA nodes (for example, 2 NUMA nodes) on each compute node by the number of compute nodes (for example, 3 compute nodes) and the executor count required per compute node (for example, 2 or 3). For this setup, the calculation 2 × 3 × (2 or 3) results in a recommended range of 12 to 18 executors.
- Calculate the total core count and determine the number of cores allocated to each executor.
Calculate the total virtual cores (for example, 288 cores) in the cluster. Determine the cores per executor based on the executor count (for example, 15). 288/15 = 19.2, which is rounded down to 18 cores because you need to reserve cores for system scheduling and the Driver program. (288 – 15 x 18 = 18 cores).
- If the CPU and memory cannot be fully loaded simultaneously, prioritize reaching full CPU load, and then determine whether to increase memory based on GC logs.
- For memory-intensive tasks, gradually adjust the memory until it is near full capacity. In this case, there may be some CPU margin, but the ratio of executor memory to cores should remain approximately equal to the ratio of total memory to total cores.