Timeout Error Caused by Improper NIC Configuration: Connect... Failed: Connection Timed Out
Symptom
When an MPI job is run on multiple nodes, the job cannot run properly, and the following error information is displayed:
[1703209660.081479] [node167:2042308:0] sock.c:272 UCX ERROR connect(fd=37, dest_addr=66.66.66.168:49703) failed: Connection timed out [node167:2042308] pml_ucx.c:426 Error: ucp_ep_create(proc=0) failed: Destination is unreachable
Possible Cause
- The firewall is not disabled. As a result, the link fails to be established.
- If the RoCE network is used, the possible cause is that the RoCE network is not configured and no IP address is specified for the network port. The show_gids command output shows that no IP address is configured for the RoCE network port.
DEV PORT INDEX GID IPv4 VeR DEV --- ---- ----- --- ------------ --- --- mlx5_0 1 0 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b8 v1 enplsOfOnp0 mlx5_0 1 1 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b8 v2 enp1s0fOnp0 mlx5_1 1 0 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b9 v1 enp1s0f1npl mlx5_1 1 1 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b9 v2 enp1s0f1npl n gids found=4
Recovery Procedure
- Disable the firewall.
- Configure the RoCE network.
DEV PORT INDEX GID IPv4 VeR DEV --- ---- ----- --- ------------ --- --- mlx5_0 1 0 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b8 v1 enplsOfOnp0 mlx5_0 1 1 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b8 v2 enp1s0fOnp0 mlx5_1 1 0 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b9 xx.xx.xx.xxx v1 enp1s0f1npl mlx5_1 1 1 fe80:0000:0000:0000:0ac0:ebff:fe8c:23b9 xx.xx.xx.xxx v2 enp1s0f1npl n gids found=4
Parent topic: FAQ