Background

High-performance computing (HPC) is a computing method and technology that uses large-scale, high-speed, and parallel computer systems to solve complex scientific, engineering, and business problems. HPC aims to accelerate problem solving and improve computation precision by using a substantial number of computing resources and parallel processing capabilities.

Despite the HPC Solution has been widely used in industries such as manufacturing, meteorology, life science, education, oil and gas, and EDA, it still faces the following challenges:

• Complex command interaction
  • Resource management is not designed from a user-centric perspective, requiring users to master some complex concepts before operation.
  • Resource management configuration is complex, making it difficult for users to master quickly and prone to errors.
• Complex industrial applications

  The HPC application industries are complex, with multiple types of applications coexisting.

• Data security

  Data is frequently transmitted. Therefore, data transmission efficiency and security need to be ensured.

• Anomaly monitoring

  Multidimensional anomalies within clusters need to be detected promptly, and multiple clusters need to be managed and monitored in a unified manner.

• Discontinuous workflows

  Conventional HPC systems are decoupled from users' existing workflows. Users need to log in to different terminals to handle data processing and transmission separately, as there is no unified portal to integrate these workflows.

• Low resource utilization

  Users work only on their own desktop workstations. This isolated working mode makes collaboration difficult and may cause low resource utilization.

• Small-scale clusters

  The clusters are small. Multiple clusters are required to support services, which increases the O&M workload and cannot support large-scale message passing interface (MPI) jobs.

• Low cluster throughput and low resource utilization

  The cluster throughput decreases as the cluster scale increases, further reducing resource utilization.

• High MPI communication overhead

  The MPI communication overhead has become increasingly prominent, emerging as a critical bottleneck for cluster computing.

To address the aforementioned challenges, a core software system for high-performance computing clusters has been developed. The system integrates proprietary Donau Portal, Donau Scheduler, and HPCKit (including Hyper MPI, KML, Kunpeng Unified Parallel Library, Hyper IO, BiSheng Compiler, and GCC for openEuler) to fully unleash computing performance.