Direct Access of cVMs to PCIe Devices on OpenStack

This section uses NVMe drives as an example to describe how cVMs directly access PCIe devices. The compute nodes must be equipped with available PCIe devices.

1. Configure device information on the compute node.
   1. Install the VFIO driver.

      modprobe vfio-pci

   2. Check the BDF number of the drive. Record it as ${BDF}.

      ll /sys/class/nvme

      

   3. Modify the /etc/nova/nova.conf file.
      1. Set passthrough_whitelist.
         • Method 1: Specify the BDF number ${BDF} and add only the device corresponding to the BDF number to the device pool.

           [pci]
           passthrough_whitelist = { "address": "${BDF}" }

         • Method 2: Specify the vendor ID (${vendor_id}) and device ID (${product_id}) to add all devices of the same type to the device pool. The following operations use this method as an example.

           [pci]
           passthrough_whitelist = { "vendor_id": "${vendor_id}", "product_id": "${product_id}" }

           You can run the following command to view the vendor and device ID. The information in the red box in the following figure is the value of [${vendor_id}:${product_id}].

           lspci -nn | grep -i nvme

           

      2. Configure pci.alias, where the alias specifies the PCI device in the flavor. The name is user-defined. Set the other parameters based on your requirements.

         [pci]
         alias = { "vendor_id":"${vendor_id}", "product_id":"${product_id}", "device_type":"type-PCI", "name":"${device_name}" }

         

         For details about how to configure OpenStack device passthrough, see the OpenStack official document https://docs.openstack.org/nova/wallaby/admin/pci-passthrough.html.

      3. After the configuration is complete on the compute node, restart the nova-compute service.

         systemctl restart openstack-nova-compute

2. Configure device information on the controller node.
   1. Modify the /etc/nova/nova.conf file.
      1. Configure filter_scheduler. Add PciPassthroughFilter to enabled_filters.

         [filter_scheduler]
         enabled_filters = ...,PciPassthroughFilter
         available_filters = nova.scheduler.filters.all_filters

      2. Configure pci.alias on the controller node as well. The configuration must be the same as that on the compute node.

         [pci]
         alias = { "vendor_id":"${vendor_id}", "product_id":"${product_id}", "device_type":"type-PCI", "name":"${device_name}" }

      3. After the configuration is complete, restart the nova-scheduler and nova-api services.

         systemctl restart openstack-nova-api.service openstack-nova-scheduler.service

3. Create a cVM for device passthrough.
   1. Create a flavor with the device passthrough attribute. In the flavor, ${device_name} indicates the name of the device to be directly accessed, which must match the preceding configuration, and ${device_num} indicates the number of devices.

      openstack flavor create cca-passthrough --vcpus 4 --ram 8192 --disk 50 \
      --property trait:HW_CPU_AARCH64_HISI_VIRTCCA=required \
      --property hw:mem_secure=true \
      --property sw:qemu_cmdline="tmm-guest,id=tmm0,num-pmu-counters=1" \
      --property "pci_passthrough:alias"="${device_name}:${device_num}"

   2. Create a cVM based on the preceding flavor.

      openstack server create --image openEuler-image --flavor cca-passthrough --network public-network pci_server