Neutron fails to bind port when launching VM
I have been running a small Packstack Stein setup with 1 Controller/Network node and 6 Compute nodes. Everything was working fine for a week or so, and then I rebooted the Controller node. After that reboot, none of the VMs I had created will bind to their DHCP addresses. Also, when I try to create a new VM, Horizon gives this error:
Error: Failed to perform requested operation on instance "demo-ubuntu-1", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance a69324fb-6d8f-48db-879e-3ef7eb662e72. Last exception: Binding failed for port d5b75470-312f-43b2-9227-b7d9addd58ed, please check neutron logs for more information.].
The Neutron server.log shows this corresponding error:
2019-07-26 20:22:25.053 6556 ERROR neutron.plugins.ml2.managers [req-b95bda3
d-b022-464e-a0c9-90d8030e5141 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e4
6fba252f1fe38fd067d - default default] Failed to bind port d5b75470-312f-43b
2-9227-b7d9addd58ed on host shp-vmc-06-man for vnic_
type normal using segments [{'network_id': '64d106db-376b-4736-9e38-f820a9fc
87e3', 'segmentation_id': 50, 'physical_network': None, 'id': 'efe44ba5-d98f
-44d3-ac40-8b33d06fe6f3', 'network_type':
More Neutron server.log info from DEBUG mode:
2019-07-29 11:34:08.910 6536 DEBUG neutron.plugins.ml2.managers [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Attempting to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 on host shp-vmc-06-man for vnic_type normal with profile bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:765
2019-07-29 11:34:08.911 6536 DEBUG neutron.plugins.ml2.managers [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Attempting to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 by drivers ovn on host shp-vmc-06-man at level 0 using segments [{'network_id': '64d106db-376b-4736-9e38-f820a9fc87e3', 'segmentation_id': 50, 'physical_network': None, 'id': 'efe44ba5-d98f-44d3-ac40-8b33d06fe6f3', 'network_type': u'geneve'}] _bind_port_level /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:792
2019-07-29 11:34:08.912 6536 DEBUG networking_ovn.ml2.mech_driver [req-07fc39e0-b191-4b8a-83e0-b7c2c8599498 66c1ccba9ff34e05bc28e503c3058f3d 4ab1c3975b3e46fba252f1fe38fd067d - default default] Refusing to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 due to no OVN chassis for host: shp-vmc-06-man bind_port /usr/lib/python2.7/site-packages/networking_ovn/ml2/mech_driver.py:634
Again, the DHCP binding was working fine until I rebooted... Any ideas on what could be wrong, or where I could start looking?
I am not running Cinder in this experiment.
Here is the output of openstack compute service list
:
[root@shp-vmc-00-man ~]# openstack compute service list
+----+------------------+----------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+----------------+----------+---------+-------+----------------------------+
| 10 | nova-conductor | shp-vmc-00-man | internal | enabled | up | 2019-07-29T14:14:59.000000 |
| 21 | nova-scheduler | shp-vmc-00-man | internal | enabled | up | 2019-07-29T14:14:59.000000 |
| 31 | nova-consoleauth | shp-vmc-00-man | internal | enabled | up | 2019-07-29T14:14:58.000000 |
| 32 | nova-compute | shp-vmc-01-man | nova | enabled | up | 2019-07-29T14:15:00.000000 |
| 33 | nova-compute | shp-vmc-02-man | nova | enabled | up | 2019-07-29T14:14:57.000000 |
| 34 | nova-compute | shp-vmc-04-man | nova | enabled | up | 2019-07-29T14:14:57.000000 |
| 35 | nova-compute | shp-vmc-05-man | nova | enabled | up | 2019-07-29T14:14:57.000000 |
| 36 | nova-compute | shp-vmc-06-man | nova | enabled | up | 2019-07-29T14:15:00.000000 |
| 37 | nova-compute | shp-vmc-03-vm | nova | enabled | up | 2019-07-29T14:15:00.000000 |
+----+------------------+----------------+----------+---------+-------+----------------------------+
Here is the output of openstack network agent list
:
[root@shp-vmc-00-man ~]# openstack network agent list
+--------------------------------------+----------------------+----------------+-------------------+-------+-------+-------------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+----------------------+----------------+-------------------+-------+-------+-------------------------------+
| 13b83a90-acd9-4c77-84ed-1f7caa2e2a5e | DHCP agent | shp-vmc-00-man | nova | XXX | UP | neutron-dhcp-agent |
| dbf1018e-4178-48d0-a360-b315790897ef | OVN Controller agent | localhost | n/a | :-) | UP | ovn-controller |
| 40dad5f7-b8d4-4c67-8376-2ba6672cf1fa | OVN Metadata agent | localhost | n/a | :-) | UP | networking-ovn-metadata-agent |
| 37564b7a-8ea7-47e1-a2d3-dbd692728f05 ...
Have you checked the other services (nova-compute, neutron) on the compute nodes? Does the controller show them as "up"? Can you show the output of
openstack compute service list
+openstack network agent list
+openstack volume service list
?Thank you for responding. I am not running Cinder (just using local LVM storage for instances in this experiment), so
openstack volume service list
returnspublic endpoint for volumev2 service in RegionOne region not found
. I will post the other lists in the edited post.Note that neutron-dhcp-agent shows its STATE ad UP, but its ALIVE as XXX.
Adding more neutron server.log info from DEBUG mode into the original post...
Refusing to bind port caa7070a-8e71-482c-87d9-a0e9deac5106 due to no OVN chassis for host: shp-vmc-06-man
indicates that something's not right regarding OVS. Can you check the respective services? I'm not sure