Ask Your Question
1

Unable to create a VM with SR-IOV port

asked 2016-02-23 00:29:09 -0500

updated 2016-02-25 00:00:56 -0500

Setup details

I have a Single Controller + Compute node (Dell Power Edge 630) with Ubuntu 14.04.4 LTS Server 64-bit and OpenStack Liberty installed.

The single controller + compute node has a Intel Corporation Ethernet 10G 2P X520 Adapter , 2 processors with 10 cores each.

From the Server BIOS, I have NUMA Topology enabled (2 NUMA nodes), HyperThreading enabled (Ubuntu sees 40 logical processors) and Virtualization Mode set to SR-IOV.

I followed the below steps

  1. Edit /etc/default/grub with below values

    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=8-39 default_hugepagesz=1G hugepagesz=1G hugepages=8 hugepagesz=2048kB hugepages=4096"

    GRUB_CMDLINE_LINUX="intel_iommu=on”

  2. “sudo update-grub” && sudo reboot

  3. Created /etc/modprobe.d/ixgbe.conf with below contents

    options ixgbe max_vfs=7

  4. “sudo update-initramfs –k all –t –u” && sudo reboot

  5. “sudo modinfo ixgbe | less” to get below driver version

    filename: /lib/modules/3.13.0-77-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

    version: 3.15.1-k

  6. “sudo modinfo ixgbevf | less"

    filename: /lib/modules/3.13.0-77-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko

    version: 2.11.3-k

  7. Executed “sudo lspci –vvv | grep –i ethernet” to ensure Virtual Functions were showing up as below

    82:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

    82:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)

    82:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

    82:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

  8. Executed “lspci –nn | grep –i Virtual” to capture PCI Vendor ID (vendor_id:product_id format). The value of interest below is 8086:10ed

    82:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

  9. Obtain the SR-IOV device name; the “ip a | less” command indicated that my SR-IOV device name was

    p1p1

  10. Verify that there are VFs assigned to the device by executing command

    “sudo ls -l /sys/class/net/p1p1/device/virtfn*"

  11. Configuration changes to nova.conf (both controller and ...

(more)
edit retag flag offensive close merge delete

Comments

Nice writeup. Not sure what the RetryFilter is. Or if the order of the filters matters. Can you try moving PciPassthroughFilter to the front of the list. And try removing all other filters as a troubleshooting step.

darragh-oreilly gravatar imagedarragh-oreilly ( 2016-02-23 01:41:07 -0500 )edit

can u try with flavor m1.small or m1.tiny?

teju gravatar imageteju ( 2016-02-23 05:26:46 -0500 )edit

Can you try running again with debug = True in /etc/nova/nova.conf (and restart the scheduler to pick it up)? What I'm after is the more detailed logging of which filters rejected hosts.

sgordon gravatar imagesgordon ( 2016-02-23 10:42:48 -0500 )edit

sgordon, appreciate your suggestion, I have now captured the logs and is at link http://paste.openstack.org/show/487923/ (log from another host setup for CPU Pinning as well )

Below is the log from the host for which I had provided the setup instuctions http://paste.openstack.org/show/487929/

vishwanathj gravatar imagevishwanathj ( 2016-02-23 12:06:11 -0500 )edit

Thanks, as I think I said on IRC it looks like the scheduler is selecting a host but then something is failing when trying to make the claims on the host so it ends up back in the RetryFilter eventually failing once the number of permitted retries (3 by default IIRC) have been used.

sgordon gravatar imagesgordon ( 2016-02-24 19:26:26 -0500 )edit

1 answer

Sort by » oldest newest most voted
2

answered 2016-02-24 23:20:58 -0500

updated 2016-02-29 15:05:45 -0500

I finally found the issue. Step 13 was missing the issue highlighted in bold below, since Intel NICS dont support sriov agent, the agent_required = False was required to be specified explicitly in ml2_conf_sriov.ini file.

Configuration changes to ml2_conf_sriov.ini

[ml2_sriov]

supported_pci_vendor_devs = 8086:10ed

agent_required = False

[sriov_nic]

physical_device_mappings = sriovnet:p1p1
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-02-23 00:29:09 -0500

Seen: 434 times

Last updated: Feb 29 '16