Preparations

Preparations

1. Modify the Hadoop configuration file on the server node.
   1. Modify ${HADOOP_HOME}/etc/hadoop/yarn-site.xml.
      1. Enable the container-executor configuration and set the user group to ockadmin. See the following figure:

         <property>
             <name>yarn.nodemanager.container-executor.class</name>
             <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
         </property>
         <property>
             <name>yarn.nodemanager.linux-container-executor.group</name>
             <value>ockadmin</value>
         </property>
         <property>
             <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
             <value>false</value>
         </property>

      2. Enable the Node Labels feature.
         Figure 1 Enabling Node Labels
         

         The following is a configuration example of yarn-site.xml:

         <property>
             <name>yarn.node-labels.enabled</name>
             <value>true</value>
         </property>
         <property>
             <name>yarn.node-labels.fs-store.root-dir</name>
             <value>/tmp/node-labels</value>
         </property>
         <property>
             <name>yarn.resourcemanager.scheduler.class</name>
             <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
         </property>

      3. Restart the ResourceManager for the configurations to take effect.
         • Query the node list to check whether the ResourceManager is running.

           yarn node -list

         • Stop the ResourceManager service.

           yarn --daemon stop resourcemanager

         • After the ResourceManager service is stopped, restart the service.

           yarn --daemon start resourcemanager

   2. Modify the ${HADOOP_HOME}/etc/hadoop/core-site.xml file. Set the proxy user and user group to ${HADOOP_USER} and ${HADOOP_GROUP} respectively.

      <property>
          <name>hadoop.proxyuser.sparkadmin.hosts</name>
          <value>*</value>
      </property>
      <property>
          <name>hadoop.proxyuser.ockadmin.groups</name>
          <value>*</value>
      </property>

   3. Add the following content to $HADOOP_HOME/etc/hadoop/container-executor.cfg. If the file does not exist, create it and set the correct file permission. For details, see the official document.

      yarn.nodemanager.linux-container-executor.group=ockadmin #configured value of yarn.nodemanager.linux-container-executor.group
      banned.users= #comma separated list of users who can not run applications
      min.user.id=1000 #Prevent other super-users
      allowed.system.users= #comma separated list of system users who CAN run applications
      feature.tc.enabled=false

   4. Perform the following operations on all nodes.
      1. Switch to the ockadmin user. Create the .hadooprc file in the user directory and change the permission to 640.

         su - ockadmin
         touch .hadooprc
         chmod 640 .hadooprc

      2. Add the following content to the file:

         export HADOOP_USER_NAME=ockadmin
         export HDFS_DATANODE_USER=ockadmin
         export HDFS_NAMENODE_USER=ockadmin
         export HDFS_SECONDARYNAMENODE_USER=ockadmin
         export YARN_RESOURCEMANAGER_USER=ockadmin
         export YARN_NODEMANAGER_USER=ockadmin

2. Restart Hadoop.
   Go to the Hadoop bin directory and restart Hadoop.

   cd $HADOOP_HOME/sbin
   ./stop-all.sh
   ./start-all.sh

   After Hadoop is restarted, run the jps command to check whether the ResourceManager is started.

   If it is not started, view the ResourceManager logs for analysis.

   If it is not started because the secure mode is enabled, exit the secure mode and restart the Hadoop service.

   hadoop dfsadmin -safemode leave
   ./start-all.sh