Yarn Working Mechanism and Scheduling

Yarn Working Mechanism and Scheduling

I. Yarn Working Mechanism

Basic Concepts:

1. Yarn does not understand the working mechanism of the programs submitted by users.

2. Yarn is only responsible for scheduling computing resources.

3. The supervisor role in Yarn is called ResourceManager.

4. The role that actually provides computing resources in Yarn is NodeManager.

Yarn main components: ResourceManager, NodeManager. NodeManager includes ApplicationMaster and container.

Main functions of RM:

1. Handle client requests.

2. Start or monitor AppMaster (inform AppMaster of idle NodeManagers).

3. Monitor NodeManager (monitor NodeManager resources).

4. Resource allocation and scheduling.

Main functions of NodeManager:

1. Manage resources on a single node.

2. Handle commands from RM.

3. Handle commands from AppMaster.

Main functions of AppMaster:

1. Responsible for data splitting (start maptask and reduce task based on the splitting results).

2. Apply for resources for the application and allocate them to internal tasks.

3. Task monitoring and fault tolerance.

Container:

Abstracts the runtime environment of tasks, encapsulating multi-dimensional resources such as CPU and memory, as well as environment variables, startup commands, and other task-related information.

Yarn working mechanism:

Workflow:

0. The MR program is submitted to the node where the client is located via the job.submit() method.

1. YarnRunner applies for an application from ResourceManager.

2. RM returns the resource path of the application (resource submission path and application_id) to YarnRunner.

3. The program submits the required resources (jar packages, configuration files, split information) to HDFS.

4. After the program resources are submitted, it applies to run an MRAppMaster (the specific implementation of AM in MR programs).

5. RM initializes the user's request into a task, which is placed in the task queue, waiting for the scheduler.

6. NodeManager picks up the task.

7. This NodeManager creates a container and starts MRAppMaster.

8. The container copies resources from HDFS to the local machine.

After MRAppMaster obtains the container, it encapsulates the task (map or reduce) related information (execution commands, environment variables required for execution commands, jars, etc.) into a ContainerLaunchContext object. The ContainerLaunchContext object and container information are then encapsulated again into a StartContainerRequest object.

9. MRAppMaster applies to RM for containers to run maptask.

10. RM assigns the maptask to two other NodeManagers, which respectively pick up the tasks and create containers.

When a task is assigned to a container by the MR scheduler, MRAppMaster starts the container by contacting the NodeManager.

11. MRAppMaster sends program startup scripts (StartContainerRequest objects) to the two NodeManagers that accepted the tasks. These two NodeManagers start maptask to partition and sort the data.

The task is executed by a Java application whose main class is YarnChild. YarnChild runs in a dedicated JVM.

12. MRAppMaster applies to RM for 2 containers to run reduce task.

13. Reduce task fetches data of the corresponding partitions from maptask.

14. After the program finishes running, MRAppMaster unregisters itself from RM.

II. Yarn Scheduler

1. Hadoop schedulers are mainly divided into three categories:

FIFO, Capacity Scheduler, and Fair Scheduler.

The default resource scheduler for Apache is the Capacity Scheduler;

The default resource scheduler for CDH is the Fair Scheduler.

2. Differences:

FIFO Scheduler: Supports a single queue, first-in-first-out. Not used in production environments.

Capacity Scheduler: Supports multiple queues. For queue resource allocation, it prioritizes allocating resources to the queue with the lowest resource usage rate; for job resource allocation, it allocates resources based on job priority and submission time order; for container resource allocation, it follows the locality principle (same node / same rack / different nodes and different racks).

Fair Scheduler: Supports multiple queues, ensuring each task fairly shares queue resources. When resources are insufficient, they can be allocated based on the deficit.

Specific configuration is in yarn-default.xml

<property>
 <description>The class to use as the resource scheduler.</description>
 <name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>