Documentation
Rate This Document
Findability
Accuracy
Completeness
Readability
SubmitClose

Using KAE on a KVM

KAE can be used on a kernel-based virtual machine (KVM). Before using KAE, ensure that VMs have been created on the host OS, KAE has been installed, and related configurations have been completed on the host OS.

The accelerator device complies with the Peripheral Component Interconnect express (PCIe) specifications, is presented as a PCIe device on the OS, and supports the single-root I/O virtualization (SR-IOV) capability. Each accelerator provides 1024 queues. A Physical Function (PF) uses 256 queues by default, and the other 768 queues are reserved for Virtual Functions (VFs). Number of VF queues = (1024 – Number of PF queues)/Number of VFs. The remainder queues are added to the last VF. It is recommended that one PF be virtualized into eight VFs.

Configuring Virtualization Settings on the Host OS

  1. Query the accelerators installed in the host OS environment and the corresponding BDF numbers.
    1
    ls -al /sys/class/uacce

  2. Configure accelerator VF settings. For example, virtualize three VFs from the hisi_sec devices each, corresponding to hisi_sec-8 to hisi_sec-13.
    1
2
    echo 3 > /sys/devices/pci0000:74/0000:74:01.0/0000:76:00.0/sriov_numvfs
echo 3 > /sys/devices/pci0000:b4/0000:b4:01.0/0000:b6:00.0/sriov_numvfs

Configuring Accelerator Settings on the VM

  1. Edit the configuration file of vm1.
    1
    virsh edit vm1
  2. Add the vCPU configuration to the configuration file. For example, configure four cores for the VM.
    1
2
3
4
5
6
7
    <cputune> 
<vcpupin vcpu='0' cpuset='4'/> 
<vcpupin vcpu='1' cpuset='5'/> 
<vcpupin vcpu='2' cpuset='6'/> 
<vcpupin vcpu='3' cpuset='7'/> 
<emulatorpin cpuset='4-7'/> 
</cputune>

    After the configuration, the VM processes run on the specified physical CPUs of the host.

  3. Configure VFs for the VM.
    • Configure one VF for the VM.
      <hostdev mode='subsystem' type='pci' managed='yes'> 
  <source> 
    <address bus='0x76' slot='0x00' function='0x1'/> 
  </source> 
</hostdev>

      After the configuration, a VF virtualized by the accelerator is mounted to the VM.

    • Configure multiple VFs for the VM.
      <hostdev mode='subsystem' type='pci' managed='yes'> 
  <source> 
    <address bus='0x76' slot='0x00' function='0x1'/> 
  </source> 
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'> 
  <source> 
    <address bus='0x76' slot='0x00' function='0x2'/> 
  </source> 
</hostdev>
    • In this section, the VF configured for the VM is virtualized by hisi_sec-8, whose BDF number is hisi_sec-8 -> ../../devices/pci0000:74/0000:74:01.0/0000:76:00.1/uacce/hisi_sec-8. The data configured in the VM XML file comes from 76:00.1 in the BDF number. 76 indicates the bus, 00 indicates the slot, and 1 indicates the function. Since the data is hexadecimal, 0x must be added.
    • In this section, only the SEC device is virtualized and configured. Therefore, after the VM is installed, only this device is available. If the VM requires a ZIP or HPRE device, add the required device by referring to the method of configuring the SEC device.
    • A hisi_sec device with the SBDF number of 0000:7x:xx.x corresponds to a device on CPU 0. The value starting with 0000:bx:xx.x corresponds to a device on CPU 1.
    • To ensure stable performance, you are advised to select the core of the corresponding CPU for the VM and select the VF virtualized by the corresponding accelerator.
    • The host OS allows a maximum of 11 VFs to be mounted to a VM.
  4. Start the VM.
    1
    virsh start vm1

    If the VM fails to be started and the message "Unknown PCI header type '127'" is displayed, unbind the mounted VFs and restart the VM.

    1
2
3
    echo 0000:76:00.1 > /sys/bus/pci/drivers/hisi_sec/unbind
echo vfio-pci > /sys/devices/pci0000:74/0000:74:01.0/0000:76:00.1/driver_override
echo 0000:76:00.1 > /sys/bus/pci/drivers_probe
  5. Log in to the VM and install KAE.

    For details, see Installing KAE.

  6. Query devices on the VM.
    1
    ls /sys/class/uacce/

    If the following information is displayed, the VF is detected on the VM.

    1
    hisi_sec-0