Fixing OpenStack: GPU Passthrough Prevents Instance Launch

When trying to enable GPU passthrough on OpenStack, you may encounter issues where instances fail to launch. This can be a complex problem due to the various components and configurations involved. Here’s a step-by-step guide to diagnose and resolve the issue:

Common Reasons for Instances Failing to Launch with GPU Passthrough

1. Misconfigured Nova Compute Service:

• The Nova compute service needs to be configured correctly to handle GPU passthrough. Incorrect settings in the nova.conf file can prevent instances from launching.

2. Incorrect or Missing PCI Passthrough Configuration:

• OpenStack needs to know which PCI devices (GPUs) are available for passthrough. This is set in the nova.conf under the [pci] section.
• Ensure that the GPU and any required devices (such as audio components often bundled with GPUs) are specified correctly.

3. Inadequate BIOS/UEFI Settings:

• The host machine’s BIOS or UEFI firmware settings must support and correctly configure IOMMU (VT-d for Intel, AMD-Vi for AMD) and SR-IOV (if needed).
• Check that the IOMMU is enabled and that the GPU is set to be visible to the operating system for passthrough.

4. Kernel and Driver Issues:

• The host operating system’s kernel and the GPU drivers must support PCI passthrough.
• Verify that the GPU is using the correct driver (such as the vfio-pci driver) and not a standard graphics driver like nouveau or nvidia.

5. Cinder or Storage Issues:

• If your instance depends on Cinder for volume storage, any misconfiguration or issues with Cinder can cause instances to fail to launch.
• Check that your storage backend is properly configured and available.

6. Resource Allocation and NUMA Topology:

• Ensure that there are enough resources (CPU, memory, and PCI slots) on the host.
• Check that the NUMA topology and resource pinning are configured correctly to support PCI passthrough.

7. Libvirt and QEMU Configuration:

• OpenStack often uses QEMU and libvirt for virtualization. Incorrect settings in the libvirt or qemu configuration files can prevent proper PCI passthrough.
• Verify the libvirt settings and ensure that the GPU is being passed through correctly.

Steps to Troubleshoot and Resolve

1. Check Nova Configuration:

• Verify that the [pci] section in nova.conf is correctly configured.
• Ensure that the pci_passthrough_whitelist and pci_alias settings include the correct vendor and product IDs for your GPU.

   [pci]
   passthrough_whitelist = {"vendor_id":"1234", "product_id":"5678"}
   alias = {"vendor_id":"1234", "product_id":"5678", "name":"gpu", "device_type":"type-PF"}

2. Verify BIOS/UEFI Settings:

• Restart the host and enter BIOS/UEFI settings.
• Enable IOMMU (Intel VT-d or AMD-Vi).
• Ensure any settings related to PCIe or device visibility are correctly configured for passthrough.

3. Update Kernel and GPU Drivers:

• Ensure that your kernel version supports IOMMU and PCI passthrough.
• Load the vfio-pci module and bind your GPU to this driver:
  bash echo "vfio-pci" > /sys/bus/pci/devices/0000:01:00.0/driver_override echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
• Replace 0000:01:00.0 with your GPU’s PCI address.

4. Inspect Libvirt and QEMU Settings:

• Check /etc/libvirt/qemu.conf for correct GPU passthrough settings.
• Ensure that the GPU is included in the VM’s XML configuration.

5. Validate Resource Availability:

• Use lscpu and lspci to check available CPUs and PCI devices.
• Ensure the host has enough free resources to accommodate the GPU and any other requirements of the instance.

6. Review Logs for Errors:

• Check the Nova and Libvirt logs for any error messages related to PCI passthrough.
  bash sudo tail -f /var/log/nova/nova-compute.log sudo tail -f /var/log/libvirt/libvirtd.log
• Look for specific errors that can give more clues about what’s going wrong.

7. Check Host and Hypervisor Compatibility:

• Ensure that your host and hypervisor software versions are compatible with GPU passthrough.
• Look up known issues or limitations in the documentation of your specific OpenStack version.

Example Configuration for GPU Passthrough

Here’s a basic example of how to set up GPU passthrough in nova.conf:

[pci]
passthrough_whitelist = [{"vendor_id": "10de", "product_id": "1db6", "address": "0000:04:00.0"}]
alias = {"vendor_id":"10de", "product_id":"1db6", "name":"nvidia_gpu"}

• Replace 10de with your GPU’s vendor ID and 1db6 with the product ID.
• Replace 0000:04:00.0 with your GPU’s PCI address.

Final Checks

1. Reboot the Host: After making changes to BIOS, kernel, or driver settings, a reboot is often necessary.
2. Test with a Simple Instance: Try launching a simple VM with minimal resources and the GPU assigned to verify basic functionality.
3. Documentation and Community: Consult the OpenStack and hardware-specific documentation, and consider asking for help in forums or communities if issues persist.

Conclusion

By checking all these reasons in a systematic way, you can identify and resolve the issues with instances failing to launch. OpenStack needs to know the PCI devices that are recognized as the best GPU dedicated server components. If you follow all the troubleshooting steps and systemically follow them, you may easily resolve the issues preventing your OpenStack instances from launching with GPU passthrough enabled.