Ask Your Question
0

Pike - SR-IOV instance - not enough hosts available [closed]

asked 2018-07-17 18:04:11 -0500

Erik Launay gravatar image

Hi

I have installed OpenStack Pike on CentOS 7 (Linux networkingnode 3.10.0-862.6.3.el7.x86_64), I can create instances, network, router, volume etc..... absolutely no issue here, everything works fine

EXCEPT if I try to create an instance with a SR-IOV port, in that case only, I'll get the error: "There are not enough hosts available".

I dedicated my Mellanox Technologies MT27520 Family [ConnectX-3 Pro] for the SR-IOV

This is how I configure the controller and compute node:

1- Enable SR-IOV in BIOS

2- Modify the Kernel with the options intel_iommu=on iommu=pt:

/etc/sysconfig/grub: 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos_computenode/root rd.lvm.lv=centos_computenode/swap rhgb quiet intel_iommu=on iommu=pt"
GRUB_DISABLE_RECOVERY="true"

and then

[root@computenode ~]# dracut --regenerate-all --force

and reboot the server

3- NIC driver installation:

[root@computenode ~]# lspci | grep Mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

[root@computenode ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

download MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.5-x86_64.tar

tar -xvf MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.5-x86_64.tar
./mlnxofedinstall 

[root@computenode]#  modprobe -rv  ib_isert rpcrdma ib_srpt
rmmod ib_isert
rmmod iscsi_target_mod
rmmod rpcrdma
rmmod ib_srpt
[root@computenode]#  /etc/init.d/openibd restart
Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]
[root@computenode MLNX_OFED_LINUX-4.4-1.0.0.0-rhel7.5-x86_64]#

reboot the server

[root@computenode ~]# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices

[root@computenode ~]# mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4103_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:04:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 00
/dev/mst/mt4103_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:04:00.0 bar=0x96400000 size=0x100000
                                   Chip revision is: 00


[root@computenode ~]# mlxconfig -d /dev/mst/mt4103_pciconf0 q

Device #1:
----------

Device type:    ConnectX3Pro
Device:         /dev/mst/mt4103_pciconf0

Configurations:                              Next Boot
         SRIOV_EN                            True(1)
         NUM_OF_VFS                          8
         LOG_BAR_SIZE                        3
         BOOT_OPTION_ROM_EN_P1               False(0)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             None(0)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               False(0)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             None(0)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
         CQ_TIMESTAMP                        True(1)

[root@computenode ~]# ibstat
CA 'mlx4_0'
        CA type: MT4103
        Number of ports: 2
        Firmware version: 2.42.5000
        Hardware version: 0
        Node GUID: 0xec0d9a0300e78930
        System image GUID: 0xec0d9a0300e78930
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x04010000
                Port GUID: 0xee0d9afffee78930
                Link layer: Ethernet
        Port 2:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x04010000
                Port GUID: 0xee0d9afffee78931
                Link layer: Ethernet

Create (or edit) /etc/modprobe.d/mlx4_core.conf

options mlx4_core num_vfs=8 port_type_array=2,2 probe_vf=0

Restart the driver

/etc/init.d/openibd restart

Check that the VFs ... (more)

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by Bernd Bausch
close date 2018-07-24 01:14:04.634563

Comments

Your scenario: The scheduler picks a host and the VM launch on that host fails. The scheduler goes through another cycle finding a host. In the second cycle, the retry filter ensures that the failing host is not selected a 2nd time.

You may get clues from the failing host’s compute log.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-17 20:45:26 -0500 )edit

2 answers

Sort by » oldest newest most voted
0

answered 2018-07-20 07:33:46 -0500

AndyW gravatar image

from my notes and OS Doc, steps to enable SR-IOV are as follows:

  1. Create Virtual Functions (Compute)
  2. Whitelist PCI devices in nova-compute (Compute) pci_passthrough_whitelist in /etc/nova/nova.conf
  3. Configure neutron-server (Controller) mechanism_drivers, firewall_drivers in /etc/neutron/plugins/ml2/ml2_conf.ini
  4. Configure nova-scheduler (Controller) PciPassthroughfilter in /etc/nova/nova.conf
  5. Enable neutron sriov-agent (Compute) in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

from your descriptions, I only see steps to do (1) so far.

see https://docs.openstack.org/mitaka/networking-guide/config-sriov.html (https://docs.openstack.org/mitaka/net...)

edit flag offensive delete link more

Comments

Hi Andy

I did all of that (as described in my explanation) However I was using: https://docs.openstack.org/neutron/pike/admin/config-sriov.html (https://docs.openstack.org/neutron/pi...) as I'm using Pike (not Mikita)

Thanks

Erik Launay gravatar imageErik Launay ( 2018-07-20 10:26:09 -0500 )edit
0

answered 2018-07-18 10:24:46 -0500

Erik Launay gravatar image

updated 2018-07-18 16:27:00 -0500

Thank you Bernd

This is what I have

On the controller node: nova-scheduler.log:

    2018-07-18 10:08:43.004 1121 INFO nova.scheduler.filters.retry_filter [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] Host [u'computenode', u'computenode'] fails.  Previously tried hosts: [[u'computenode', u'computenode']]
    2018-07-18 10:08:43.004 1121 INFO nova.filters [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] Filter RetryFilter returned 0 hosts
    2018-07-18 10:08:43.004 1121 INFO nova.filters [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] Filtering removed all hosts for the request with instance ID '9b693626-633e-4933-af28-7556bd03e629'. Filter results: ['RetryFilter: (start: 1, end: 0)']

* On the compute node *

nova-compute.log:

2018-07-18 10:08:38.526 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] Total memory: 327458 MB, used: 512.00 MB
2018-07-18 10:08:38.526 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] memory limit: 491187.00 MB, free: 490675.00 MB
2018-07-18 10:08:38.527 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] Total disk: 5559 GB, used: 0.00 GB
2018-07-18 10:08:38.527 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] disk limit not specified, defaulting to unlimited
2018-07-18 10:08:38.528 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] Total vcpu: 80 VCPU, used: 0.00 VCPU
2018-07-18 10:08:38.529 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] vcpu limit not specified, defaulting to unlimited
2018-07-18 10:08:38.532 2296 INFO nova.compute.claims [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] Claim successful on node computenode
2018-07-18 10:08:39.238 2296 INFO nova.virt.libvirt.driver [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] [instance: 9b693626-633e-4933-af28-7556bd03e629] Creating image
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager [req-1b0acf05-debe-467e-a81c-3df78ad4c3c3 409a679ba4c840eeb46b12768c6ef60a a72a5d6b06d14b63acec9774146b0f6e - default default] Instance failed network setup after 1 attempt(s): PortBindingFailed: Binding failed for port de0870fe-8cc6-4533-a5a8-9071cfa880a0, please check neutron logs for more information.
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager Traceback (most recent call last):
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1415, in _allocate_network_async
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager     bind_host_id=bind_host_id)
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 888, in allocate_for_instance
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager     bind_host_id, dhcp_options, available_macs)
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1014, in _update_ports_for_instance
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager     vif.destroy()
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager     self.force_reraise()
2018-07-18 10:08:41.646 2296 ERROR nova.compute.manager   File "/usr ...
(more)
edit flag offensive delete link more

Comments

Unfortunately I don't know enough about SRIOV and how OpenStack manages SRIOV devices to really help. "Binding" means connecting a port to the physical infrastructure, which includes connecting the VM to the network. It seems to fail because Nova somehow doesn't see the VFs.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-18 18:30:55 -0500 )edit

Having said that, I think what ultimately leads to the failure is a lookup of /sys/bus/pci/devices/PCI_ADDRESS/net. I wonder if your config contains an incorrect PCI address, e.g. missing leading zeros or too many leading zeros or so.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-18 18:40:16 -0500 )edit

In case you wonder how I know this, I took the error message and found that it was probably generated by _get_sysfs_netdev_path, called by get_ifname_by_pci_address, called by get_net_name_by_vf_pci_address.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-18 18:43:00 -0500 )edit

The Nova-Compute log tries to find devices with IP addresses 0000:04:00.X. Check if they exist under /sys/bus/pci/devices, point to an existing directory, and have a subdirectory named net. If yes, I am probably wrong. If not, you have to find out why.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-18 18:51:54 -0500 )edit

Another possibility: Those /sys directories exist but don't contain MAC addresses. In this case, get_mac_by_pci_address fails. But that should generate another error message that I don't see.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-18 18:58:50 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2018-07-17 18:04:11 -0500

Seen: 323 times

Last updated: Jul 20 '18