(Optional) Installing the KNEM Component

KNEM enhances intra-node MPI communication. Installing and enabling this component offers performance gains for large-packet scenarios in MPI communication across certain HPC applications.

Checking the KNEM Enabling Status

Before installing the KNEM component, check whether it is enabled. If it is enabled, you do not need to install it.

Run the cat /dev/knem command. If the following information is displayed, the component has been enabled:

knem 1.1.4.90 (git20250207.4b4ebc6)
 Driver ABI=0xd
 Flags: forcing 0x0, ignoring 0x0
 DMAEngine: NoKernelSupport
 Debug: NotBuilt
 Requests submitted                           : 16320
 Requests processed (total)                   : 16320
          processed (using DMA)               : 0
          processed (offloaded to thread)     : 0
          processed (with pinned local pages) : 0
 Requests rejected (invalid flags)            : 0
          rejected (not enough memory)        : 0
          rejected (invalid ioctl argument)   : 0
          rejected (unexisting region cookie) : 0
          rejected (failed to pin local pages): 0
 Requests failed during memcpy from/to user   : 0
          failed during DMA copy              : 0
 DMA copy cleanup timeout                     : 0

If KNEM is not enabled, install KNEM and use the preceding method to check whether KNEM is enabled.

Installing KNEM

KNEM must be installed on each compute node.

Download the RPM package. The RPM package varies with the OS.

openEuler 20.03 LTS AArch64

wget --no-check-certificate https://repo.oepkgs.net/openeuler/rpm/openEuler-20.03-LTS-SP1/extras/aarch64/Packages/k/knem-1.1.4.90-0.aarch64.rpm

openEuler 22.03 LTS AArch64

wget --no-check-certificate https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS/extras/aarch64/Packages/k/knem-1.1.4.90-0.aarch64.rpm

Kylin Linux Advanced Server V10 (HPC) AArch64

wget --no-check-certificate https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS/extras/aarch64/Packages/k/knem-1.1.4.90-0.ky10h.aarch64.rpm

Kylin Linux Advanced Server V10 SP2 AArch64

wget --no-check-certificate https://repo.oepkgs.net/openeuler/rpm/openEuler-20.03-LTS-SP1/extras/aarch64/Packages/k/knem-1.1.4.90-0.ky10.aarch64.rpm

Kylinsec OS Linux release 3.5.2 AArch64

wget --no-check-certificate https://repo.oepkgs.net/openeuler/rpm/openEuler-22.03-LTS/extras/aarch64/Packages/k/knem-1.1.4.90-0.aarch64.rpm

Install the RPM package.
```
rpm -ivh rpm_name
```
If the command output indicates that the system has a built-in RPM package and the new RPM package cannot be installed, manually uninstall the built-in RPM package and then reinstall the new RPM package.
Check whether the knem kernel module has been loaded. If knem is displayed in the command output, the knem kernel has been successfully loaded.
lsmod | grep knem
1
knem 57344 4

Run the cat /dev/knem command. If the following information is displayed, the installation is successful:

knem 1.1.4.90 (git20250207.4b4ebc6)
 Driver ABI=0xd
 Flags: forcing 0x0, ignoring 0x0
 DMAEngine: NoKernelSupport
 Debug: NotBuilt
 Requests submitted                           : 16320
 Requests processed (total)                   : 16320
          processed (using DMA)               : 0
          processed (offloaded to thread)     : 0
          processed (with pinned local pages) : 0
 Requests rejected (invalid flags)            : 0
          rejected (not enough memory)        : 0
          rejected (invalid ioctl argument)   : 0
          rejected (unexisting region cookie) : 0
          rejected (failed to pin local pages): 0
 Requests failed during memcpy from/to user   : 0
          failed during DMA copy              : 0
 DMA copy cleanup timeout                     : 0

Run the built-in test case knem_pingpong of KNEM. By default, the test case is installed in /opt/knem-1.1.4.90.

cd /path/to/install

/path/to/install/bin/knem_pingpong

/path/to/install indicates the KNEM installation directory.

1024:	  25.705 us	39.84 MB/s	 37.99 MiB/s
2048:	  25.869 us	79.17 MB/s	 75.50 MiB/s
4096:	  29.301 us	139.79 MB/s	 133.32 MiB/s
8192:	  30.902 us	265.09 MB/s	 252.81 MiB/s
16384:	  35.213 us	465.28 MB/s	 443.73 MiB/s
32768:	  45.170 us	725.44 MB/s	 691.83 MiB/s
65536:	  69.500 us	942.96 MB/s	 899.28 MiB/s
131072:	  62.795 us	2087.30 MB/s	 1990.61 MiB/s
262144:	  91.160 us	2875.64 MB/s	 2742.43 MiB/s
524288:	  158.172 us	3314.67 MB/s	 3161.12 MiB/s
1048576:  216.383 us	4845.93 MB/s	 4621.44 MiB/s
2097152:  310.549 us	6753.05 MB/s	 6440.21 MiB/s
4194304:  329.018 us	12747.96 MB/s	 12157.40 MiB/s
8388608:  615.082 us	13638.19 MB/s	 13006.40 MiB/s
16777216: 1232.287 us	13614.70 MB/s	 12983.99 MiB/s

Parent topic: Appendixes