Timeout Error Due to ARP Entry Not Found: ibv_create_ah...failed: Connection timed out
Symptom
When a user submits large-scale MPI jobs, there is a high probability that a connection setup timeout error is reported and the job output log contains "Connection timed out".
[agent363:373418:0:373418] ud iface.c:49 Fatal: iface 0x1ddfcb30: failed to get peer address === backtrace (tid: 373418) ==== 0 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucs.so.0(ucs_fatal_error_message+0x38) [0x40012c420128] 1 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucs.so.0(+0x2025c) [0x40012c42025c] 2 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/ucx/libuct_ib.so.0(uct_ud_iface_cep_insert_ep+0) [0x40012c5157e0] 3 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/ucx/libuct_ib.so.0(uct_ud_ep_create_connected_common+0xd4) [0x40012c5184c4] 4 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucp.so.0(ucp_wireup_ep_connect_aux+0xc0) [0x400127f51be0] 5 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucp.so.0(ucp_wireup_ep_connect+0xe4)[0x400127f522e4] 6 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucp.so.0(ucp_wireup_init_lanes+0x8d4)[0x400127f53e94] 7 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucp.so.0(ucp_ep_create_to_worker_addr+0x78) [0x400127f1cf58] 8 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hucx/lib/libucp.so.0(ucp_ep_create+0x4b0) [0x400127f1dbbe] 9 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/openmpi/mca_pml_ucx.so(+0x5940)[0x400127e55940] 10 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/openmpi/mca_pm1_ucx.so(mca_pml_ucx_send+0x198) [0x400127e54304] 11 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/libmpi.so.40(ompi_coll_base_sendrecv_actual+0xbc)[0x4001266ffafc] 12 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/libmpi.so.40(ompi_coll_base_sendrecv_intra_bruck+0xac)[0x4001266fe710] 13 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/openmpi/mca_coll_ucx.so(+0x5b08) [0x40012cb25b08] 14 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/libmpi.so.40(mca_coll_base_comm_select+0x880)[0x4001266f3d64] 15 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/libmpi.so.40(ompi_mpi_init+0xe30) [0x40012672a7a4] 16 /share/software/mpi/hmpi/1.2.1/bisheng2.5.0/hmpi/lib/libmpi.so.40(MPI_Init+0xa8) [0x4001266d8468] 17 hello_mpi() [0x4008c0] 18 /usr/lib64/libc.so.6(__libc_start_main+0xe0) [0x400126813f40] 19 hello_mpi() [0x4007dc] ================================
The job error log contains " failed to get peer address".
[1693797506.380997] [agent289:3924149:0] ib_device.c:1252 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=::ffff:192.168.1.115 sgid index=5 traffic class=106) on mlx5 0 failed: Connection timed out [1693797506.384196] [agent276:1958414:0] ib_device.c:1252 UCX ERROR ibv_create_ah(dlid=49152 sl=0 port=1 src_path_bits=0 dgid=::ffff:192.168.1.62 sgid_index=5 traffic_class=106) on mlx5_0 failed: Connection timed out
Possible Cause
- The error is reported by the ibv_create_ah UCX module of MPI. The cause is that the peer address is not obtained within the timeout interval.
- The switch's ARP attack defense feature is configured by default to limit the number of ARP packets sent from a single source IP address to 30 per second. When large-scale Hyper MPI jobs (more than 100 nodes) are started, more than 30 ARP packets are sent within 1 second (without ARP cache). As a result, ARP packets are lost, and the MPI link establishment times out.
- After the peer address is cached locally by using the local ARP cache table of the Linux, the peer address can be obtained from the table, and the ARP request does not need to be sent again.
Recovery Procedure
Configure static ARP entries on the compute node.
Parent topic: FAQ