An Error Is Reported After a GID Is Specified in the -x UCX_IB_GID_INDEX Command

Symptom

An error is reported after an invalid GID is specified in -x UCX_IB_GID_INDEX.

ib_device.c:848 UCX ERROR ibv_query_gid(dev=mlx5_0 port=1 index=10) failed: No such file or directory

An error is reported after a GID is specified in -x UCX_IB_GID_INDEX.

pml_ucx.c:384  Error: ucp_ep_create(proc=1) failed: Destination is unreachable

Possible Cause

• The GID specified in the CLI is different from the GID of the server.
• The GIDs of multiple job execution nodes are inconsistent.

Recovery Procedure

• The GID specified in the CLI is different from the GID of the server.
  1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
  2. Run the following command to query the GID value:

     show_gids | grep -E "((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])" | grep v2 | awk '{print $3}'

     5

5 indicates the queried GID value.

1. Run the following command to set the GID to the GID queried in 2, for example, 5:

   -x UCX_IB_GID_INDEX=5

• The GIDs of multiple job execution nodes are inconsistent.
  1. Use PuTTY to log in to a job execution node as a common Hyper MPI user, for example, hmpi_user.
  2. Modify the ~/.bashrc file.

     Open the ~/.bashrc file.

     vi ~/.bashrc

     Press i to enter insert mode and add the following content:

     v2_gid=$(show_gids | grep -E "((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])" | grep v2 | awk '{print $3}') 
     export UCX_IB_GID_INDEX=$v2_gid

     Press Esc, type :wq!, and press Enter to save the settings and exit.

  3. Run the following command for the settings to take effect:

     source ~/.bashrc