Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

OpenStack (Rocky): VM not spawning on provider network though it is successfully getting spawned on self-service network.

Hello Everyone, I am trying to set-up OpenStack (Rocky) cloud using the standard guide (Available at link:https://docs.openstack.org/install-guide/) for OpenStack (Rocky) installation. Below here are the details provided. Please let me know if any further details are required, I shall be very happy to provide.

Cloud Setup

My setup contains one controller node and two compute nodes. Both the nodes contain 2 NIC. One for management network (10.0.0.0/24 - NIC name enp4s0) and other for provider network (10.20.4.0/22 - enp2s0).
Controller node: All the services are functional and running successfully on the controller node. systemctl status SERVICE shows all the services required for successful working of controller node are up and running. Neutron (networking service) is also running on the controller node.
Compute nodes: All the services are functional and running successfully on the compute node. systemctl status SERVICE shows all the services required for successful working of compute node up and running.

I have two networks created. 1. Provider network (flat with external provider network - 10.20.4.0/22) 2. Self-service network (192.168.0.0/24)

There already is 1 OpenStack cloud running on the same physical network (The physical network I am using for the provider network) if that can be a problem which I think is very unlikely. Both the setups are using separate IP pools.

Problem

Instances are successfully getting created on the Self-service network. But when I create an instance on Provider network, instance gets with an error "Failed to allocate the network(s), not rescheduling". After trying to solve the problem, I think (Just a guess though :-) ) I have figured out the problem.

The problem is with the bridge that is getting created on the physical provider network interface (brqeedc5776-03 on enp2s0) on both compute nodes. The same bridge that that is getting created on the physical provider network interface on controller is working fine. Physical provider network interface (enp2s0) gets added to the bridge automatically on controller node and is working fine whereas on the compute node it is missing as can be seen in the following screenshots.

image:/home/abdul/brctl-show-controller.xcf Physical interface 'enp2s0' already added to the bridge-brqeedc5776-03 (This bridge is being used for provider network).

image:/home/abdul/brctl-show-compute01.xcf VXLAN interface successfully added to the bridge and 1 instance successfully running on that bridge-brqab88d4ca-68 (This bridge is being used for Self-service network).

Solution that did not help

I tried adding the physical interface to the bridge-brqeedc5776-03 (This bridge is being used for provider network) manually using brctl addif brqeedc5776-03 enp2s0 on compute node. But it did not solve the problem and I am still getting the same error.

Log files: I am not getting any error in log files on controller node. The only error I am getting is in the logfile on path /var/log/neutron/neutron-linuxbridge-agent.log on the controller node. I get this error only on the controller node on which the VM(served by provider network) gets scheduled. Following is the error log. I am spawning the instance as admin and still getting the permission denied error.

2019-04-23 16:31:42.213 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-4342b879-a822-4c06-9882-ad06c04b1660 - - - - -] Error in agent loop. Devices info: {'current': set(['tap807a015e-9c']), 'timestamps': {'tap807a015e-9c': 10}, 'removed': set([]), 'added': set(['tap807a015e-9c']), 'updated': set([])}: NetlinkError: (13, 'Permission denied')
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 458, in daemon_loop
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(args, *kwargs)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 209, in process_network_devices
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent resync_a = self.treat_devices_added_updated(devices_added_updated)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(args, *kwargs)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 226, in treat_devices_added_updated
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._process_device_if_exists(device_details)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 253, in _process_device_if_exists
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent device, device_details['device_owner'])
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 582, in plug_interface
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent network_segment.mtu)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 520, in add_tap_interface
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent return False
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.force_reraise()
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent six.reraise(self.type_, self.value, self.tb)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 512, in add_tap_interface
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent tap_device_name, device_owner, mtu)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 544, in _add_tap_interface
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent mtu):
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 496, in ensure_physical_in_bridge
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent physical_interface)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 285, in ensure_flat_bridge
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent if self.ensure_bridge(bridge_name, physical_interface):
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 454, in ensure_bridge
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.update_interface_ip_details(bridge_name, interface)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 416, in update_interface_ip_details
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent gateway)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 393, in _update_interface_ip_details
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent dst_device.addr.add(cidr=ip['cidr'])
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 604, in add
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent add_broadcast)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 995, in add_ip_address
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent device, namespace, scope, broadcast)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent return self.channel.remote_call(name, args, kwargs)
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 202, in remote_call
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent raise exc_type(*result[2])
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent NetlinkError: (13, 'Permission denied')
2019-04-23 16:32:39.324 1751 ERROR neutron.plugins.ml2.drivers.agent._common_agent