Failed to Start Hadoop Using Yarn

Symptom

The following information is displayed when using Yarn to start Hadoop 3.2.2 or later:

2024-09-19 15:41:42,256 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state INITED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
...
Caused by: ExitCodeException exitCode=24: File /home/sparkadmin/hadoop-3.2.0/etc/hadoop/container-executor.cfg must be owned by root, but is owned by 1002

Key Process and Cause Analysis

LinuxContainerExecutor uses container-executor to start containers. For system security purposes, the owner of the container-executor.cfg file and its parent paths must be the root user. The path configured in container-executor is the default path and container-executor must be recompiled.

Conclusion and Solution

1. Obtain the Hadoop source package of the corresponding version (Hadoop 3.2.0-RC1 is used as an example): https://github.com/apache/hadoop/releases/tag/release-3.2.0-RC1.
2. Decompress the source package and go to the hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager directory.
3. Perform the compilation.

   cmake src -DHADOOP_CONF_DIR=/etc/hadoop
   make

4. Go to the target/native/target/usr/local/bin directory and extract the container-executor file and change the file owner to root. Replace the corresponding binary file in the system environment.