Key Features
Hyper IO has the following key features:
- Aggregates, combines, and reorganizes datasets across multiple dimensions based on the physical meaning of application data in different scenarios, reducing metadata volume and I/O access frequency.
- Identifies data types and statistics of typical HPC applications and applies differentiated data compression technologies to different data types (including floating-point numbers, integers, and sparse matrices) to save network transmission bandwidth and improve data interaction efficiency and stability.
- Splits application data processing steps, decouples algorithms of processing units in different phases, and flexibly calls processing units based on data attributes to implement data-driven processing.
- Supports pre- and post-processing operations of any amount, type, and sequence on a single data object and aggregation, disassembling, and reorganization operations in any form and dimension on multiple data objects.
- Senses and analyzes dataset information of applications to formulate and execute optimization policies.
The current Hyper IO version bridges and accelerates NetCDF, PnetCDF, HDF5, and MPI-IO libraries, with support for POSIX I/O semantics coming in a future release. The following sections detail the APIs for each supported I/O library.
Parent topic: Product Overview