Using the Terminal Tool to Analyze MPI Application Performance

Function

Uses the terminal tool to analyze MPI application performance.

Check that the System Profiler has been installed. The default tool directory is /usr/bin/devkit/mpi_adaptive.

**Table 1** Subcommands
Subcommand	Description
sclrun	Collects common call information of MPI applications.
sclread	Reads the collected performance data and converts it into a visual file.
time_line	Analyzes the collected function information, saves the mpi_send and mpi_recv event information to the database, collects statistics on the time series data of events, and aligns the startup event timeline of each MPI process.
query_timeline_data	Queries the event information about the MPI processes.
-h/--help	Obtains help information.

Syntax

      
           ./mpi_adaptive_tool [-h | --help] COMMAND [ARGS]

The Open MPI version must be 4.0.3 or later.
The tool can only on the Kunpeng 920 platform.
Add the -g parameter when compiling the application to be collected. Otherwise, the variable and function symbol information may fail to be obtained when the tool collects call stack information, and the error is indicated using question marks (??) in the result report.
To analyze a multi-node cluster, copy the /usr/bin/devkit/mpi_adaptive directory to the shared directory of the cluster. Then, modify the logPath variable in the mpi_adaptive/config/cfg.ini file in the shared directory based on your requirements.

sclrun subcommand:

       
            mpi_adaptive_tool sclrun [ -h | --help ] [ --backtrace ] [ -backtrace_max_depth <depth> ] [ -samp_mode {1,2,3} ] [ -samp_ratio <ratio> ] [ -samp_random <seed> ] [ -l | --log-level {0,1,2,3} ] [ -o | --output <collect_result_directory> ] [ <exe> <args> ]

[<exe> <args>] indicates the MPI application and parameters to be collected. It is mandatory when you run the command. Enter it at the end of the command line.

sclread subcommand:

       
            mpi_adaptive_tool sclread [ -h | --help ] [ -i | --input <sclrun_result_dir> ] [ -o | --output <readable_output_file> ] [ -l | --log-level {0,1,2,3} ] [ -f | --force ]

time_line subcommand:

       
            mpi_adaptive_tool time_line [ -h | --help ] [ -i | --input <sclrun_result_dir> ] [ -o | --output <analyze_output_file> ] [ -l | --log-level {0,1,2,3} ] [ -f | --force ]

query_timeline_data subcommand:

       
            mpi_adaptive_tool query_timeline_data [ -h | --help ] [ --path <timeline_db_dir> ] [ -p | --page <page_number> ] [ -s | --pagesize <page_size> ] [ -r | --rank <rank_number> ] [ -d | --detail ]

Subcommand Parameters

sclrun: collects common call information of MPI applications.

**Table 2** **sclrun** subcommand parameters
Command	Parameter	Description
-h/--help	-	Obtains help information.
-o/--output	-	(Mandatory) Output directory of the collected application performance data.
-l/--log-level	0/1/2/3	Log level, which defaults to 1. The log configuration file is in the config/cfg.ini directory. 0: DEBUG 1: INFO 2: WARNING 3: ERROR
--backtrace	-	Collects the call stack information of applications.
-backtrace_max_depth	-	Maximum depth of the call stack to be collected, which defaults to 20 layers. This parameter is configurable when --backtrace is used.
-samp_mode	1/2/3	Sampling mode for collecting performance data. The default value is 1. 1: Full sampling 2: Fixed sampling ratio. Data is collected based on the sampling ratio specified by the --samp_ratio parameter. 3: Adaptive sampling. Data is collected based on the sampling ratio determined by the number of ranks.
-samp_ratio	-	Sampling ratio, which ranges from 0 to 100 and defaults to 0. This parameter is configurable when -samp_mode is set to 2.
-samp_random	<seed>	Enables random sampling based on the specified random seed, which is an integer. Only the randomness of the rank values is controlled and the sampled number of ranks is not changed.

sclread: reads the collected performance data and converts it into a visual file.

**Table 3** **sclread** subcommand parameters
Command	Parameter	Description
-h/--help	-	Obtains help information.
-l/--log-level	0/1/2/3	Log level, which defaults to 1. The log configuration file is in the config/cfg.ini directory. 0: DEBUG 1: INFO 2: WARNING 3: ERROR
-i/--input	<sclrun_result_dir>	(Mandatory) Directory for storing collected performance data result files.
-o/--output	-	Output directory of the visual report.
-f/--force	-	If the output directory already exists, the sclread content is forcibly overwritten.

time_line: aligns the startup event timeline of each MPI process.

**Table 4** **time_line** subcommand parameters
Command	Parameter	Description
-h/--help	-	Obtains help information.
-l/--log-level	0/1/2/3	Log level, which defaults to 1. The log configuration file is in the config/cfg.ini directory. 0: DEBUG 1: INFO 2: WARNING 3: ERROR
-i/--input	<sclrun_result_dir>	(Mandatory) Directory for storing collected performance data result files.
-o/--output	-	Directory for storing the result file of the time offset of each event. This file is an SQLite database file and is named time_series.db by default.
-f/--force	-	If the output directory already exists, the time_line content is forcibly overwritten.

query_timeline_data: queries the event information about the MPI processes.

**Table 5** **query_timeline_data** subcommand parameters
Command	Parameter	Description
-h/--help	-	Obtains help information.
--path	<timeline_db_dir>	Path to the database file to be queried.
-p/--page	-	Page number for data query, which defaults to 1.
-s/--pagesize	-	Number of data records displayed on each page, which defaults to 10.
-r/--rank	-	Filters a specified rank. If no rank is specified, all data is displayed by default.
-d/--detail	-	Displays detailed call stack information. If this option is disabled, only brief call stack information is displayed by default.

Example

Switch to the tool directory.
1

cd /usr/bin/devkit/
When the System Profiler is installed, the default tool directory is /usr/bin/devkit/mpi_adaptive.

Copy the mpi_adaptive directory to the shared directory.

To run the mpirun command, run the tool in the shared directory.

        
             cp -r mpi_adaptive /home/share
cd /home/share/mpi_adaptive

Collect the application performance data.

        
             /opt/bin/mpirun -x PATH -x LD_LIBRARY_PATH -mca btl ^openib --mca pml ucx -bind-to core  -np 10 --map-by ppr:96:node ./mpi_adaptive_tool sclrun -o /home/share/tmp --backtrace -- /home/mpi_adaptive_tool/test_mpi

/opt/bin/mpirun -x PATH -x LD_LIBRARY_PATH -mca btl ^openib --mca pml ucx -bind-to core -np 10 --map-by ppr:96:node is the command of running MPI. -o /home/share/tmp indicates that the collection result is outputted to the /home/share/tmp directory, which must be in the shared directory. --backtrace indicates that the call stack information is collected. -- /home/mpi_adaptive_tool/test_mpi specifies the application whose information needs to be collected. The file generated after the collection is a binary file.

Convert the performance data into a visual file.

        
             ./mpi_adaptive_tool sclread  -i /home/share/tmp -o /home/tmp2 -f

-i /home/share/tmp indicates that the performance data collected by sclrun is stored in the /home/share/tmp directory. -o /home/tmp2 indicates that the performance data is converted into a visual file and is stored in the /home/tmp2 directory. -f indicates that the existing sclread content in the specified directory is forcibly overwritten. The generated text file is readable.

Command output:

        
             Configured: 
	Input: /home/share/tmp
	Output: /home/tmp2
=== Event 19 ===
Event Name: MPI_Recv
Start Time: 717638437792265
End Time: 717638437792333
Duration: 68
Record Size: 58
Data type: 4326592
Dest/Src: 0
Type Size: 4
Tag: 20
Count: 1
MPI Comm: 4325568

Align the MPI application event timeline.

        
             ./mpi_adaptive_tool time_line -i /home/share/tmp -o /home/time_line.db -f

-o /home/time_line.db specifies the name of and path to the generated db file. If the db file name is not specified, a time_series.db file is generated in the current directory by default. The file can be viewed in the SQLite database. -f indicates that the existing time_line content in the specified directory is forcibly overwritten.

Command output:

        
             Configured: 
	Input: /tmp/
	Output: /home/time_line.db

View the data information of the MPI application.

        
             ./mpi_adaptive_tool query_timeline_data --path  /home/time_line.db -d

--path /home/time_line.db specifies the db file to be queried, and -d displays the detailed call stack information.

Figure 1 Details

**Table 6** Output of **query_timeline_data**
Parameter	Description
id	MPI message ID.
rank	Rank ID.
event	Event type.
start_time	System counter value when the MPI starts to run.
end_time	System counter value when the MPI stops running.
use_time	System counter change value during the MPI running. The value can be converted into a duration in seconds using the *use_time / (freq_mhz 10^6) formula, where freq_mhz is the clock resolution in MHz and can be obtained by running the sudo dmesg \| grep clock** command.
datatype	Data type.
typesize	Size of the data of the corresponding type.
dest	Rank that communicates with the current rank. The value MPI_Send indicates the rank that receives messages and MPI_Recv indicates the rank that sends messages.
tag	Message identifier.
count	Number of sent or received MPI messages.
comm	Communicator.
backtrace	Call stack information.

Parent topic: Using Basic Functions in the Terminal Tool