Error "UCX ERROR Failed to Create SDMA_Device[0] Handle" Reported When UCX Uses SDMA

Symptom

When UCX uses SDMA, the following error is reported:

UCX ERROR Failed to create sdma_device[0] handle

Possible Cause

In SMMU-enabled environments, the sm transport layer of UCX uses SDMA as the shared memory for data transmission by default. The SDMA kernel driver in a single I/O die manages 160 dedicated and shared channels. Each time MPI starts a service process, UCX requests an SDMA shared channel on the I/O die hosting that process. If no SDMA shared channel is requested, MPI reports an error and exits during initialization. If -x UCX_SDMA_SHARED_CHANNEL_MODE=n is set to use dedicated channels but the number of requested SDMA dedicated channels is insufficient, MPI reports an error and exits during initialization.

Recovery Procedure

Before using SDMA to run services, check whether the number of SDMA shared channels in a single I/O die is greater than 0. To use dedicated channels, manually change the number of SDMA dedicated channels based on the service needs. By default, the total number of SDMA shared and dedicated channels in a single I/O die is 160. Therefore, you can adjust the number of shared channels to change the number of dedicated ones. The change method is as follows:

Check the number of SDMA shared channels in a single I/O die.
cat /sys/module/sdma_dae/parameters/share_chns
```
16
```
Remove the SDMA kernel module.
rmmod sdma_dae
Re-insert the SDMA kernel module and change the number of shared channels.
- Example 1:
  modprobe sdma_dae share_chns=160
  
  Since the number of shared channels is changed to 160, the number of dedicated channels is 0. Therefore, this configuration allows the MPI to start SDMA service processes in a single I/O die.
- Example 2
  modprobe sdma_dae share_chns=8
  
  Since the number of shared channels is changed to 8, the number of dedicated channels is 152. Therefore, this configuration allows the MPI to start up to 152 service processes in a single I/O die though -x UCX_SDMA_SHARED_CHANNEL_MODE=n.

Parent topic: FAQ