Feature Description

Overview

Hyper IO is a middleware layer that bridges system and application software. It provides storage description format definitions and file operations for scientific data, and is widely deployed in high-performance computing (HPC) applications, post-processing, visualization, and analytics frameworks.

Benefits

To enhance application scalability and end-to-end performance for HPC, Huawei integrates its proprietary Hyper IO into its HPC solution. This Kunpeng-optimized middleware provides comprehensive compatibility with multiple open-source middleware APIs. By abstracting and classifying applications through the optimization of inter-process and inter-step I/O paradigms, Hyper IO coalesces, reorganizes, and caches data blocks. It reshapes application I/O into file-system-friendly operations, eliminating I/O bottlenecks for large-scale applications and accelerating runtime performance on the Kunpeng platform.

Functions

Hyper IO enhances the scalability and end-to-end performance of large-scale, I/O-intensive HPC applications and reduces parallel file system failures, resolving I/O bottlenecks. It addresses the sustainability issues inherent in applications running storage-specific optimizations.

• Review the I/O library APIs used in applications before enabling Hyper IO. Unsupported APIs may cause undefined behavior. Exercise caution when using Hyper IO acceleration features.
• Since the underlying native APIs of the open-source middleware lack built-in thread safety, the APIs intercepted and wrapped by Hyper IO inherit this non-thread-safe behavior.
• Ensure the validity of input parameters passed to bridging functions. All API behavior conforms strictly to the original specification.
• Specify the .so files used by the middleware, including libock_shore_mpiio.so, libock_shore_pnetcdf.so, libock_shore_netcdf.so, and libock_shore_hdf5.so in an environment variable. Ensure the security, availability, and reliability of the .so files.
• Ensure that the LD_PRELOAD environment variable is allowed in the user scenario.
• The APIs comply with the MPI-IO, PnetCDF, NetCDF, and HDF5 standards. Callers must ensure the parameter validity, coding security, and penetration safety.

Application Scenarios

Hyper IO is widely used in HPC applications such as LAMMPS, GRAPES, NEMO, and WRF, as well as a large number of post-processing, visualization, and analysis frameworks.