Understanding YARN Cluster Mode

1. Executing a script to submit a job actually starts a SparkSubmit JVM process; the main method in the SparkSubmit class reflectively calls the main method of YarnClusterApplication, which creates a Yarn client and then sends an execution command to the Yarn server: bin/java ApplicationMaster;

2. After receiving the command, the Yarn framework starts ApplicationMaster in the specified NM;

3. ApplicationMaster starts the Driver thread to execute the user's job;

4. AM registers with RM and requests resources;

5. RM returns a list of available resources;

6. After obtaining resources, AM sends a command to NM: bin/java YarnCoarseGrainedExecutorBackend;

7. The CoarseGrainedExecutorBackend process receives messages, communicates with the Driver, and registers the started Executor; then starts the computation object Executor to wait for tasks;

8. The Driver thread continues to complete job scheduling and task execution.

9. The Driver assigns tasks and monitors their execution.

Note: SparkSubmit, ApplicationMaster, and CoarseGrainedExecutorBackend are independent processes; Driver is an independent thread; Executor and YarnClusterApplication are objects.