---
title: 多节点运行mpirun时异常
description: "- 多节点运行mpirun命令时，无任何反应。运行top命令时，无mpirun进程。"
url: https://www.hikunpeng.com/document/detail/zh/kunpenghpcs/hpckit/devg/userg_huaweimpi_0040.html
sourcePath: /source/zh/kunpenghpcs/hpckit/devg/userg_huaweimpi_0040.html
indexId: 347def71b54269b5a4a6baf60d5ec03aa064fb884bb554f95b37555f232c0ede60
---
# 多节点运行mpirun时异常

#### 现象描述

- 多节点运行mpirun命令时，无任何反应。运行top命令时，无mpirun进程。
- 多节点运行mpirun命令时，出现如下报错信息：

```
[1632387881.405868] [arm-node88:57923:0]       mm_posix.c:194  UCX  ERROR shm_open(file_name=/ucx_shm_posix_23f3f65f flags=0xc2) failed: Permission denied
[1632387881.405910] [arm-node88:57923:0]        uct_mem.c:132  UCX  ERROR failed to allocate 8447 bytes using md posix for mm_recv_fifo: Shared memory error
[1632387881.405917] [arm-node88:57923:0]       mm_iface.c:605  UCX  ERROR mm_iface failed to allocate receive FIFO
[arm-node88:57923] coll_ucx_component.c:360  Warning: Failed to create UCG worker, automatically select other available and highest priority collective component.
[1632387881.411347] [arm-node88:57923:0]       mm_posix.c:194  UCX  ERROR shm_open(file_name=/ucx_shm_posix_6ae5143e flags=0xc2) failed: Permission denied
[1632387881.411359] [arm-node88:57923:0]        uct_mem.c:132  UCX  ERROR failed to allocate 8447 bytes using md posix for mm_recv_fifo: Shared memory error
[1632387881.411366] [arm-node88:57923:0]       mm_iface.c:605  UCX ERROR mm_iface failed to allocate receive FIFO
[arm-node88:57923] pml_ucx.c:274  Error: Failed to create UCP worker
[arm-node88:57923] *** An error occurred in MPI_Allreduce
[arm-node88:57923] *** reported by process [878510081,70368744177671]
[arm-node88:57923] *** on communicator MPI_COMM_WORLD
[arm-node88:57923] *** MPI_ERR_INTERN: internal error
[arm-node88:57923] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[arm-node88:57923] ***    and potentially your MPI job)
```


#### 可能原因

多节点运行mpirun命令时，存在节点与节点间不能互相通信。


#### 恢复步骤

1. 使用PuTTY工具，以Hyper MPI普通用户（例如“hmpi_user”）登录至作业执行节点。
2. 检查所有节点是否都安装Hyper MPI且各节点安装路径是否一致，或者检查Hyper MPI是否安装在已挂载的共享目录上。
3. 检查环境变量是否配置正确。详情请参见《Kunpeng HPCKit 26.1.RC1 安装指南》(https://www.hikunpeng.com/document/detail/zh/kunpenghpcs/hpckit/instg/KunpengHPCKit_install_024.html)中“安装卸载HPCKit > 设置环境变量”。