Using Hyper IO

Prerequisites

Hyper IO and third-party libraries (such as MPI, PnetCDF, NetCDF, and HDF5) required for application running have been installed.

IO Middleware cannot be used together with AdhocFS or I/O Forward of Hyper IO.

Mappings

Procedure

1. Use PuTTY to log in to the job execution node as a common user.
2. Run the following command to set the environment variable for the JSON configuration file:

   export OCK_IO_CONFIG=/opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/conf/xxx.json

   For example, run the export OCK_IO_CONFIG=/opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/conf/grapes.json command for GRAPES.

   

   The current user must be the owner of the JSON configuration file and have the read permission.

3. (Optional) Modify the JSON configuration file for optimal application performance.
   

   • JSON sample files are provided for different applications. You can choose one that suits your application or modify the file as required.
   • Do not set duplicate configuration items in the configuration file. If duplicate configuration items exist, the first one takes precedence. For example, if "LOG_FLOW" : "ON" and "LOG_FLOW" : "OFF" appear in sequence, the value ON takes effect.
   • To modify a sample file, copy its content into a new JSON file, and then modify it as required.

   1. Run the following commands to copy and configure a JSON file:

      cp /opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/conf/xxx.json /userpath/to/xxx.json

      vi /userpath/to/xxx.json

   2. Press i to enter the insert mode and modify the configuration file as required. See parameter description in Table 2, Table 3, and Table 4.
      • (Optional) Configure the global dataset information.

        Table 2 Global dataset information (DATASET)

        Item		Description	Type	Default Value
        Global dataset general information (GENERAL)	LOG_FLOW	Prints FLOW module logs.	String ON: enable OFF: disable	OFF
        	LOG_PROCESSOR	Prints PROCESSOR module logs.	String ON: enable OFF: disable	OFF
        	LOG_BRIDGE	Prints BRIDGE module logs.	String ON: enable OFF: disable	OFF
        	LOG_INTERCEPTOR	Prints logs of all INTERCEPTOR modules.	String ON: enable OFF: disable	OFF
        	LOG_CONFIG	Prints CONFIG module logs.	String ON: enable OFF: disable	OFF
        	TRANSCACHESIZE	Aggregates data into a single write operation when the total data volume reaches the cache size. The value range is [0, 128]. The unit is MB. The default value is 0, indicating that aggregation is not performed.	Integer	0
        	COMPRESSMETHOD	Specifies the data compression method. The value range is [1, 2]. The default value is 1.	Integer 1: LZ4 compression algorithm 2: Bzip2 compression algorithm	1
        	COMPRESSLEVEL	Specifies the data compression level. The value range is [0, 5]. The default value is 0, indicating that compression is not performed. A higher data compression level indicates a higher compression ratio.	Integer	0
        	WITHOUTGROUP	Connects to Hyper IO without configuring group information.	String ON: enable OFF: disable	OFF
        Dataset access pattern information (DATA_ACCESS_PATTERN)	FIXED	Indicates whether write operations of all entities remain unchanged between steps.	Boolean false true	false
        	RANDOM	Indicates whether to randomize the particle order in particle applications.	Boolean false true	false
        	WRITE_AGGREGATE_STEPS	Specifies the number of metadata steps to aggregate into a single write, mitigating frequent small I/O operations. The value range is [1, 64].	Integer	1
        	WRITE_AGGREGATE_RANKS	Specifies the number of metadata ranks to aggregate into a single write. The value range is [1, 64].	Integer	1

      • (Optional) To use the file aggregation function, configure the global dataset group information.

        Table 3 Global dataset group information (GROUPS)

        Name	Description	Type	Default Value
        GROUPNAME	Mandatory. Name of the aggregated file in aggregation mode. The value must match the regular expression specified for FILEWHITELIST.	String of up to 64 characters	-
        FILEWHITELIST	Optional. Regular expression to match files to be aggregated into the GROUPNAME file.	String of up to 64 characters	-

      • (Optional) To enable out-of-order output for a single file, configure the dataset entity information.

        Table 4 Dataset entity information (ENTITIES)

        Name	Description	Type	Default Value
        NAME	Mandatory. Name of the entity represented by a path.	String of up to 64 characters	-
        SHUFFLE	Optional. Whether to randomize the particle order in particle applications.	Boolean false true The value is case-insensitive.	false

      • The following configuration file sample contains all configuration items. Include only the required items in your configuration file.

        1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

        { "DATASET": { "GENERAL": { "LOG_FLOW" : "OFF", "LOG_PROCESSOR": "OFF", "LOG_BRIDGE" : "OFF", "LOG_INTERCEPTOR" : "OFF", "LOG_CONFIG" : "OFF", "TRANSCACHESIZE" : 0, "COMPRESSMETHOD" : 1, "COMPRESSLEVEL" : 0, "WITHOUTGROUP" : OFF, "DATA_ACCESS_PATTERN": { "FIXED": "false", "RANDOM": false, "WRITE_AGGREGATE_STEPS": 1, "WRITE_AGGREGATE_RANKS": 1 } }, "GROUPS": [ { "GROUPNAME": "xxxxx", "FILEWHITELIST" : "xxxxx", "ENTITIES": [ { "NAME": "xxxxx", "SHUFFLE": false } ] } ] } }

   3. Press Esc, type :wq!, and press Enter to save the settings and exit.
   4. Set the environment variable for the JSON configuration file.

      export OCK_IO_CONFIG=/userpath/to/xxx.json

4. Run the application to utilize the I/O acceleration capability. Run the application with the LD_PRELOAD environment variable configured to redirect I/O operations to Hyper IO. Upon completion, the execution will generate specific output artifacts, including directories suffixed with shore and format, as well as a file bearing the same name as the original.
   • The directory suffixed with shore is used to store data information.
   • The directory suffixed with format is used to store structure information that describes the data.
   • The files with the same names are used to store the I/O Middleware configuration file information.

   Take Grapes as an example. Run mpirun -N 80 -x LD_PRELOAD=/opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/lib/libock_shore_mpiio.so -x OMP_NUM_THREADS=1 -mca pml ucx -mca btl ^vader,tcp,openib,uct -x UCX_TLS=self,sm,rc -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_WARN_UNUSED_ENV_VARS=0 ./grapes_global.exe. After the command is executed, a directory suffixed with shore and a file with the same name as the original file are generated.

   

   • /opt/IO_Middleware/IO_Middleware_2.6.1_aarch64 indicates the installation directory of Hyper IO.
   • Only one acceleration library environment variable can be set for each HPC application.
5. (Optional) Use the format conversion function of Hyper IO.
   

   • Before using the format conversion function of Hyper IO, ensure that the open-source MPI-IO has been installed in the system environment. Otherwise, the function may not run properly.
   • When using the format conversion function of Hyper IO, ensure that the target path for conversion is different from the output path for the conversion result.
   • If the conversion fails, logs are displayed on the screen. You need to manually delete the generated files.
   • To set multiple threads, add -map-by ppr: m:node:pe=n to the mpirun command for resource mapping. m indicates the number of processes running on each node, and n indicates the number of CPU cores available for each process.
   • Recommended application configurations:
     • If the bcc-esm application uses the NetCDF library, the conversion tool does not require -t and -d.
     • If the WRF application uses the PnetCDF library, the conversion tool requires -t8 and -d.

   To use the format conversion function of Hyper IO, perform the following steps:

   • Single-node conversion

     /opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/bin/ock_shore_convertor [--format <Target_format_(only_netcdf_is_supported_currently)>] [-t <1-8> (number_of_threads)] [-d (No parameter is required. Configure whether to enable dataset compression. By default, dataset compression is disabled.)] Shared_storage_directory_(shore_file_or_file_directory) Output_directory_of_the_target_file

   • Multi-node and multi-process conversion

     mpirun -hostfile $HOSTFILE -np Number_of_processes /opt/IO_Middleware/IO_Middleware_2.6.1_aarch64/bin/ock_shore_convertor [--format <Target_format_(only_netcdf_is_supported_currently)>] [-t <1-8> (number_of_threads)] [-d (No parameter is required. Configure whether to enable dataset compression. By default, dataset compression is disabled.)] Shared_storage_directory_(shore_file_or_file_directory) Output_directory_of_the_target_file

6. (Optional) If Hyper IO is no longer required, run the following command to delete the environment variable in effect:

   unset OCK_IO_CONFIG

Applications and Versions Supported by Hyper IO