OpenStack VMs Failed to Start After the Physical Machine Is Powered Off and Powered On
Symptom
In the scenario where OpenStack is integrated with Ceph, after a physical machine is forcibly (or unexpectedly) powered off and then powered on again, the VM cannot be started properly.
Key Process and Cause Analysis
- Check the console log during VM startup. The following error information is displayed:
Figure 1 VM startup error
- Check the log information of the VM in the /var/log/libvirt/qemu/ directory. The following error information is displayed:
Figure 2 libvirt error log
In this case, OpenStack is running properly, and the newly provisioned VM is also normal. Based on the log analysis, the VM that is unexpectedly powered off fails to connect to the backend block storage.
Conclusion and Solution
During OpenStack O&M, unexpected power-off must be avoided. If a VM has to be powered off, notify the tenants and VM users in advance, and then stop the VM before the power-off.
If an unexpected power-off occurs, perform the following steps to restore the VM:
- Search for the VM instance name.
nova show $vm_id
vm_id indicates the VM ID, which can be obtained by running the nova list command.
Figure 3 VM information
- Record instance_name of the VM.
- Obtain the backend storage information of the VM.
virsh dumpxml $instance_name|grep -w source
Figure 4 VM backend storage information
Locate the source tag where protocol='rbd'. Record the value of name and use it as rbd_name in the follow-up operations.
instance_name indicates the VM instance name.
- Mount the volume of the backend storage.
rbd map $rbd_name rbd unmap $rbd_name
- Restart to restore the VM.
nova reboot $vm_id
Wait until the VM is restarted.