Rate This Document
Findability
Accuracy
Completeness
Readability

HPC Debugger Functions

High-performance computing (HPC) is a computing method and technology that uses large-scale, high-speed, and parallel computer systems to solve complex scientific, engineering, and business problems. HPC aims to accelerate problem solving and improve computation precision by using a substantial number of computing resources and parallel processing capabilities.

The HPC Debugger supports parallel debugging on multiple nodes in HPC scenarios. MPI and MPI+OpenMP applications must be debugged in launch or attach mode.

Prerequisites

  • You have installed the HPC Debugger. See Installing the Tool.
  • If you have installed the tool using a compressed package, decompress the package and switch to the tool directory. Then run the command in ./ mode, for example, ./devkit debugger -h. If you have installed the tool using an RPM package, run devkit debugger -h. This section uses an RPM package as an example.
  • The certificate required for debugging the HPC program has been installed. For details, see Installing the Certificate.

Command Function

Debugs MPI and MPI+OpenMP applications in Launch mode. It must be run using the mpirun command.

Debugs MPI and MPI+OpenMP applications in Attach mode. It must use the srun method of the Slurm scheduler.

In Launch mode, the debugger is directly started to manage the entire lifecycle of the program. This mode applies to full-cycle debugging in local development. In Attach mode, the debugger is attached to the running process in real time. This mode applies to debugging scenarios where the debugger cannot be directly started or dynamic intervention is required.

This tool supports C, C++, and Fortran.

Syntax

Launch mode:

1
debugger launch [-h | --help] {-w APP_WORKLOAD | --workload APP_WORKLOAD} {-s SOURCE_PATH | --source SOURCE_PATH} {-m MPIRUN_CMD | --mpicmd MPIRUN_CMD} {-p RPC_SERVER_PORT | --port RPC_SERVER_PORT} [-l {0,1,2,3} | --log-level {0,1,2,3}] [-e ENVIRONMENT_VARIABLES | --env ENVIRONMENT_VARIABLES] [-a APP_ARGS | --args APP_ARGS] [-n THREAD_NUM | --threads THREAD_NUM]

Attach mode:

1
debugger attach [-h | --help] {-w APP_WORKLOAD | --workload APP_WORKLOAD} {-s SOURCE_PATH | --source SOURCE_PATH}{-r RUN_MODE | --run-mode RUN_MODE}{-j JOB_ID | --job JOB_ID} {-p RPC_SERVER_PORT | --port RPC_SERVER_PORT} [-l {0,1,2,3} | --log-level {0,1,2,3}] [-e ENVIRONMENT_VARIABLES | --env ENVIRONMENT_VARIABLES]

Parameter Description

Table 1 Launch mode parameters

Parameter

Option

Description

-h/--help

-

Obtains help information. This parameter is optional.

-w/--workload

-

Path to the executable file of the application to be debugged. This parameter is mandatory.

Example: /home/test/mpi_demo

-s/--source

-

Source file directory of the application to be debugged. This parameter is mandatory.

Example: /home/test

-m/--mpicmd

-

mpirun command to be executed. This parameter is mandatory and must be enclosed in double quotation marks.

Example: mpirun --allow-run-as-root -np 4

-p/--port

-

Port for running the agent server to report startup information. This parameter is mandatory.

-l/--log-level

0/1/2/3

Log level, which defaults to 1. This parameter is optional.

  • 0: DEBUG
  • 1: INFO
  • 2: WARNING
  • 3: ERROR

-e/--env

-

Environment variable. Use double quotation marks to enclose the environment variable. This parameter is optional. The options are:

  • export PATH=$PATH:/path/to/mpi
  • source /configure/mpi/path/file
  • module load /mpi/modulefiles

Example: export PATH=$PATH:/path/to/mpi

-a/--args

-

Running argument of the application to be debugged. If there are multiple arguments, separate them with spaces. This parameter is optional.

-n/--threads

-

Number of OpenMP threads, which ranges from 1 to 1024. This parameter is optional.

Table 2 Attach mode parameters

Parameter

Option

Description

-h/--help

-

Obtains help information. This parameter is optional.

-w/--workload

-

Path to the executable file of the application to be debugged. This parameter is mandatory.

Example: /home/test/mpi_demo

-s/--source

-

Source file directory of the application to be debugged. This parameter is mandatory.

Example: /home/test

-r/--run-mode

-

Application running mode, which must be the Slurm scheduler. This parameter is mandatory.

  • slurm

-j/--job

-

Job ID. This parameter is mandatory.

-p/--port

-

Port for running the agent server to report startup information. This parameter is mandatory.

-l/--log-level

0/1/2/3

Log level, which defaults to 1. This parameter is optional.

  • 0: DEBUG
  • 1: INFO
  • 2: WARNING
  • 3: ERROR

-e/--env

-

Environment variable. Use double quotation marks to enclose the environment variable. This parameter is optional. The options are:

  • export PATH=$PATH:/path/to/mpi
  • source /configure/mpi/path/file
  • module load /mpi/modulefiles

Example: export PATH=$PATH:/path/to/mpi

Installing the Certificate

Before using the debugger, install the tool and then install the certificate. The certificate ensures secure communication between nodes. After the installation is complete, the debugger is ready for use. As an example, install an RPM package to use the tool:

  1. Install the RPM package.
    1
    rpm -ivh devkit-x.x.x-1.aarch64.rpm devkit-debugger-x.x.x-1.aarch64.rpm 
    

    Command output:

    1
    2
    3
    4
    5
    6
    Preparing...                          ################################# [100%]
    Updating / installing...
       1:devkit-x.x.x-1                  ################################# [ 50%]
    devkit installed
       2:devkit-debugger-x.x.x-1         ################################# [100%]
    devkit-debugger installed
    
  2. Check whether the installation is successful.
    1
    rpm -qa | grep devkit
    

    The installation is successful if the installation package name is included in the command output.

    1
    2
    devkit-debugger-x.x.x-1.aarch64
    devkit-x.x.x-1.aarch64
    
  3. Make the automatic command line completion take effect.
    Log in to the terminal again or run the following command on the terminal:
    1
    source /etc/bash_completion.d/devkit.sh
    
  4. Install the certificate.
    1
    2
    cd /usr/local/devkit/debugger/
    ./install_rpc_cert
    
    • The default tool installation path is /usr/local/.
    • If the tool is installed using a TAR package, the install_rpc_cert executable file is stored in the /Path_to_DevKit_CLI/debugger/ directory. /Path_to_DevKit_CLI/ indicates the DevKit command line tool path.
  5. Configure IP addresses for communication.
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    === IP Selection ===
    
    Available IP addresses:
    1: xx.xx.xx.xx
    2: xx.xx.xx.xx
    3: xx.xx.xx.xx
    
    Select IP [default 1]: 
    
    Selected IP: xx.xx.xx.xx
    The agent rpc certificate generated successfully.
    

    When configuring the IP address, select an IP address from Available IP addresses in the command output. You can also press Enter to select the first IP address.

    After the installation is complete, the rpc_cert folder is generated in the current directory.

  6. View the directory structure of rpc_cert.
    1
    ll rpc_cert
    

    Directory structure:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    rpc_cert/
    ├── client.key             # Client key (encrypted)
    ├── client_passwd          # Password for decrypting the client key
       ├── common
       └── zeus
    ├── debugger_ca.pem       # Root certificate
    ├── debugger_client.pem     # Client certificate
    ├── debugger_server.pem    # Server certificate
    ├── server.key             
    └── server_passwd
        ├── common
        └── zeus
    

Example

  1. Compile mpi_demo.c.
    1
    2
    cd /home/test/
    mpicc -g -o mpi_demo mpi_demo.c
    

    After the compilation is complete, the mpi_demo executable file is generated.

  2. Debug the HPC application (using the /home/test/mpi_demo file as an example).
    • Start the application in Launch mode:
      1
      devkit debugger launch -w /home/test/mpi_demo -s /home/test/ -m "mpirun --allow-run-as-root -np 4" -p 9982
      
    • Start the application in Attach mode:
      devkit debugger attach -w /home/test/mpi_demo -s /home/test/ -r slurm -j <job ID> -p 9982

    After the application starts, the interactive interface of the HPC debugger is displayed. For details about debugging operations, see Debugging Status Commands and Debugging Operation Commands.