Neutron fails to bind port when launching VM

asked 2019-07-26 19:42:55 -0500

toddleish gravatar image

updated 2019-07-29 10:57:37 -0500

I have been running a small Packstack Stein setup with 1 Controller/Network node and 6 Compute nodes. Everything was working fine for a week or so, and then I rebooted the Controller node. After that reboot, none of the VMs I had created will bind to their DHCP addresses. Also, when I try to create a new VM, Horizon gives this error:

Error: Failed to perform requested operation on instance "demo-ubuntu-1", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance a69324fb-6d8f-48db-879e-3ef7eb662e72. Last exception: Binding failed for port d5b75470-312f-43b2-9227-b7d9addd58ed, please check neutron logs for more information.].

The Neutron server.log shows this corresponding error:

2019-07-26 20:22:25.053 6556 ERROR neutron.plugins.ml2.managers [req-b95bda3

d-b022-464e-a0c9-90d8030e5141 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e4

6fba252f1fe38fd067d - default default] Failed to bind port d5b75470-312f-43b

2-9227-b7d9addd58ed on host shp-vmc-06-man for vnic_

type normal using segments [{'network_id': '64d106db-376b-4736-9e38-f820a9fc

87e3', 'segmentation_id': 50, 'physical_network': None, 'id': 'efe44ba5-d98f

-44d3-ac40-8b33d06fe6f3', 'network_type':

More Neutron server.log info from DEBUG mode:

2019-07-29 11:34:08.910 6536 DEBUG neutron.plugins.ml2.managers [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Attempting to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 on host shp-vmc-06-man for vnic_type normal with profile  bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:765
2019-07-29 11:34:08.911 6536 DEBUG neutron.plugins.ml2.managers [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Attempting to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 by drivers ovn on host shp-vmc-06-man at level 0 using segments [{'network_id': '64d106db-376b-4736-9e38-f820a9fc87e3', 'segmentation_id': 50, 'physical_network': None, 'id': 'efe44ba5-d98f-44d3-ac40-8b33d06fe6f3', 'network_type': u'geneve'}] _bind_port_level /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:792
2019-07-29 11:34:08.912 6536 DEBUG networking_ovn.ml2.mech_driver [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Refusing to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 due to no OVN chassis for host: shp-vmc-06-man bind_port /usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py:634

Again, the DHCP binding was working fine until I rebooted... Any ideas on what could be wrong, or where I could start looking?

I am not running Cinder in this experiment.

Here is the output of openstack compute service list:

[root@shp-vmc-00-man ~]# openstack compute service list
+----+------------------+----------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host           | Zone     | Status  | State | Updated At                 |
+----+------------------+----------------+----------+---------+-------+----------------------------+
| 10 | nova-conductor   | shp-vmc-00-man | internal | enabled | up    | 2019-07-29T14:14:59.000000 |
| 21 | nova-scheduler   | shp-vmc-00-man | internal | enabled | up    | 2019-07-29T14:14:59.000000 |
| 31 | nova-consoleauth | shp-vmc-00-man | internal | enabled | up    | 2019-07-29T14:14:58.000000 |
| 32 | nova-compute     | shp-vmc-01-man | nova     | enabled | up    | 2019-07-29T14:15:00.000000 |
| 33 | nova-compute     | shp-vmc-02-man | nova     | enabled | up    | 2019-07-29T14:14:57.000000 |
| 34 | nova-compute     | shp-vmc-04-man | nova     | enabled | up    | 2019-07-29T14:14:57.000000 |
| 35 | nova-compute     | shp-vmc-05-man | nova     | enabled | up    | 2019-07-29T14:14:57.000000 |
| 36 | nova-compute     | shp-vmc-06-man | nova     | enabled | up    | 2019-07-29T14:15:00.000000 |
| 37 | nova-compute     | shp-vmc-03-vm  | nova     | enabled | up    | 2019-07-29T14:15:00.000000 |
+----+------------------+----------------+----------+---------+-------+----------------------------+

Here is the output of openstack network agent list:

[root@shp-vmc-00-man ~]# openstack network agent list

+--------------------------------------+----------------------+----------------+-------------------+-------+-------+-------------------------------+
| ID                                   | Agent Type           | Host           | Availability Zone | Alive | State | Binary                        |
+--------------------------------------+----------------------+----------------+-------------------+-------+-------+-------------------------------+
| 13b83a90-acd9-4c77-84ed-1f7caa2e2a5e | DHCP agent           | shp-vmc-00-man | nova              | XXX   | UP    | neutron-dhcp-agent            |
| dbf1018e-4178-48d0-a360-b315790897ef | OVN Controller agent | localhost      | n/a               | :-)   | UP    | ovn-controller                |
| 40dad5f7-b8d4-4c67-8376-2ba6672cf1fa | OVN Metadata agent   | localhost      | n/a               | :-)   | UP    | networking-ovn-metadata-agent |
| 37564b7a-8ea7-47e1-a2d3-dbd692728f05 ...
(more)
edit retag flag offensive close merge delete

Comments

Have you checked the other services (nova-compute, neutron) on the compute nodes? Does the controller show them as "up"? Can you show the output of openstack compute service list + openstack network agent list + openstack volume service list?

eblock gravatar imageeblock ( 2019-07-29 01:55:29 -0500 )edit

Thank you for responding. I am not running Cinder (just using local LVM storage for instances in this experiment), so openstack volume service list returns public endpoint for volumev2 service in RegionOne region not found. I will post the other lists in the edited post.

toddleish gravatar imagetoddleish ( 2019-07-29 09:22:19 -0500 )edit

Note that neutron-dhcp-agent shows its STATE ad UP, but its ALIVE as XXX.

toddleish gravatar imagetoddleish ( 2019-07-29 09:34:39 -0500 )edit

Adding more neutron server.log info from DEBUG mode into the original post...

toddleish gravatar imagetoddleish ( 2019-07-29 10:55:59 -0500 )edit
1

Refusing to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 due to no OVN chassis for host: shp-vmc-06-man indicates that something's not right regarding OVS. Can you check the respective services? I'm not sure

eblock gravatar imageeblock ( 2019-07-30 07:00:14 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-07-30 12:20:30 -0500

toddleish gravatar image

I was able to fix this with the help of @eblock. OVS was pulling the server hostname from a different source than I had it configured (the CentOS host file)... To fix the problem I set the hostname for each server in /etc/sysconfig/network. OVS now picks up the correct hostname and does not use localhost. I should have done this before. Thank you for the great ideas!

edit flag offensive delete link more

Comments

Great that you resolved it!

eblock gravatar imageeblock ( 2019-07-31 04:58:18 -0500 )edit

Hi, can you explain where you changed the the hostname in /etc/sysconfig/network.

Auntu gravatar imageAuntu ( 2020-01-21 10:16:26 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-07-26 19:42:55 -0500

Seen: 643 times

Last updated: Jul 29 '19