Modifying Flink Task Parameters Submitted by Yarn
Purpose
Before running a Flink application, run the yarn-session.sh -n 4 -s 64 -jm 5000 -tm 50000 -d command to submit a Flink task to apply for memory and CPU resources from Yarn. You can modify the parameters for submitting tasks to adjust concurrent tasks and resource allocation.
Procedure
Parameter |
Recommended Value |
Description |
|---|---|---|
-n (taskmanager) |
Number of nodes x (4 to 8) |
This parameter indicates the number of TaskManagers of Flink. The Flink engine requires one JobManager and multiple TaskManagers for running. Each TaskManager is independent. To run a Flink application, it is randomly allocated to a TaskManager for running. Therefore, when the number of TaskManagers increases, more Flink applications can run concurrently without interference. However, the resources in the cluster are limited. The more TaskManagers, the less resources can be allocated to a single TaskManager, which causes a decrease in Flink performance. The number of TaskManagers can be set to the number of cluster nodes x 4 to the number of nodes x 8. |
-s (slot) |
Set this parameter based on the site requirements. |
This parameter indicates the number of slots in a single TaskManager. After a Flink task is submitted, the total number of slots available in the cluster is the number of slots in a single TaskManager x the number of TaskManagers. One slot can provide one concurrent task for the TaskManager. For example, if the number of slots is 30, one Taskmanager can run a maximum of 30 concurrent tasks. In practice, you can run only 20 concurrent tasks. In this case, the remaining 10 slots are idle. The unused slots do not occupy CPU resources but occupy related memory resources. The value of this parameter is dynamically adjusted based on the concurrency required. If the memory is sufficient, set this parameter to a large value. |
-tm(taskmanager memory) |
30000 |
This parameter indicates the memory resources allocated to a single TaskManager. As long as the TaskManager memory is sufficient, the performance is not directly improved when the more memory resources are allocated. You can allocate all the remaining memory to TaskManager after the memory is allocated to JobManagers. You are advised to allocate about 30,000 MB. |
-jm(jobmanager memory) |
5000 |
This parameter indicates the memory resources allocated to JobManagers. This parameter has little impact on the overall performance and does not need to be set to a large value. You are advised to allocate 5000 MB to 15,000 MB. |