Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Failed to bind port on host

Hi, just made a fresh installation of the Liberty release on Ubuntu 14.04 and I am running into some network problem that I dont understand them. I think the biggest problem is: Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02

Services are running as seen:

neutron agent-list

+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+
| 0a1eb1ea-ec5d-4edd-b2eb-f8adf8545365 | L3 agent           | controller02 | :-)   | True           | neutron-l3-agent          |
| 0cb6076a-135a-42d6-8d73-4bd59f1dbffe | Metadata agent     | controller01 | :-)   | True           | neutron-metadata-agent    |
| 505c0d2d-15f9-4948-9e16-e9a9b0a01d87 | Linux bridge agent | controller02 | :-)   | True           | neutron-linuxbridge-agent |
| 637dc6d3-3c26-42c9-86a3-0246aa357c4f | DHCP agent         | controller02 | :-)   | True           | neutron-dhcp-agent        |
| adadaaa9-bce5-477e-a734-73a3c08f4fb8 | DHCP agent         | controller01 | :-)   | True           | neutron-dhcp-agent        |
| b7985827-ac34-4f38-9245-dc1767c3c008 | Linux bridge agent | hv03         | :-)   | True           | neutron-linuxbridge-agent |
| ef78d68b-20b5-4ea1-804e-f7befa8e2634 | Linux bridge agent | controller01 | :-)   | True           | neutron-linuxbridge-agent |
| f2038985-9f83-47e8-969d-2e91e04e02b7 | Metadata agent     | controller02 | :-)   | True           | neutron-metadata-agent    |
| f82682d9-018d-4cc4-9f94-8302e17be17a | L3 agent           | controller01 | :-)   | True           | neutron-l3-agent          |
| f8f419fc-38d9-470d-925e-fcd45778e32d | Linux bridge agent | hv01         | :-)   | True           | neutron-linuxbridge-agent |
| ff70f7cf-8c74-4642-b14c-32a2a23d91a9 | Linux bridge agent | hv02         | :-)   | True           | neutron-linuxbridge-agent |
+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+

neutron net-list

+--------------------------------------+----------+------------------------------------------------------+
| id                                   | name     | subnets                                              |
+--------------------------------------+----------+------------------------------------------------------+
| 78bba7fe-a27d-4000-bd28-704660d9c680 | Internal | d68bb402-da64-42db-897b-51f94e725e3f 172.16.101.0/24 |
+--------------------------------------+----------+------------------------------------------------------+

nova service-list

+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-cert        | controller01 | internal | enabled | up    | 2016-06-01T15:15:47.000000 | -               |
| 2  | nova-cert        | controller02 | internal | enabled | up    | 2016-06-01T15:15:51.000000 | -               |
| 3  | nova-consoleauth | controller01 | internal | enabled | up    | 2016-06-01T15:15:51.000000 | -               |
| 4  | nova-consoleauth | controller02 | internal | enabled | down  | 2016-06-01T12:38:49.000000 | -               |
| 5  | nova-scheduler   | controller01 | internal | enabled | up    | 2016-06-01T15:15:49.000000 | -               |
| 6  | nova-scheduler   | controller02 | internal | enabled | up    | 2016-06-01T15:15:52.000000 | -               |
| 7  | nova-conductor   | controller01 | internal | enabled | up    | 2016-06-01T15:15:53.000000 | -               |
| 8  | nova-conductor   | controller02 | internal | enabled | up    | 2016-06-01T15:15:49.000000 | -               |
| 9  | nova-compute     | hv01-k58         | nova     | enabled | up    | 2016-06-01T15:15:48.000000 | -               |
| 10 | nova-compute     | hv02-k58         | nova     | enabled | up    | 2016-06-01T15:15:52.000000 | -               |
| 11 | nova-compute     | hv03-k58         | nova     | enabled | up    | 2016-06-01T15:15:50.000000 | -               |
+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

When I start a machine I get following error messages:

neutron-server.log (controller node):

[...]
2016-06-01 16:24:10.164 9575 ERROR neutron.plugins.ml2.managers [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02
2016-06-01 16:24:10.164 9575 ERROR neutron.plugins.ml2.managers [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02
2016-06-01 16:24:10.165 9575 INFO neutron.plugins.ml2.plugin [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Attempt 2 to bind port 802862a2-223f-4e32-bc9c-b5821351f70b
[...]

nova-conductor.log (controller node):

[...]
2016-06-01 16:24:12.142 1656 ERROR nova.scheduler.utils [req-ec201652-17bc-486c-b5be-7846112d692b d7d8a3b5814540a4b01da3412db9c49c 20f9af47fde64253a5975e072e09ab2e - - -] [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Error from last host: hv02 (node hv02): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2058, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 1a0145bb-55f6-4ea9-9667-9e96876ea17e was re-scheduled: Binding failed for port 802862a2-223f-4e32-bc9c-b5821351f70b, please check neutron logs for more information.\n']
[...]

nova-compute.log (compute node):

[...]
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [req-ec201652-17bc-486c-b5be-7846112d692b d7d8a3b5814540a4b01da3412db9c49c 20f9af47fde64253a5975e072e09ab2e - - -] [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Instance failed to spawn
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Traceback (most recent call last):
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     yield resources
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     block_device_info=block_device_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2528, in spawn
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     write_to_disk=True)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4401, in _get_guest_xml
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     network_info_str = str(network_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 515, in __str__
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     return self._sync_wrapper(fn, *args, **kwargs)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 498, in _sync_wrapper
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 530, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self[:] = self._gt.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     return self._exit_event.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     current.throw(*self._exc)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     result = function(*args, **kwargs)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1581, in _allocate_network_async
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     six.reraise(*exc_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1564, in _allocate_network_async
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     dhcp_options=dhcp_options)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 744, in allocate_for_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self._delete_ports(neutron, instance, created_port_ids)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     six.reraise(self.type_, self.value, self.tb)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 736, in allocate_for_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     security_group_ids, available_macs, dhcp_opts)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 359, in _create_port
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     raise exception.PortBindingFailed(port_id=port_id)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] PortBindingFailed: Binding failed for port 84255ab0-9576-43a5-b175-a75295a4fe60, please check neutron logs for more information.
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] 
[...]

I dont see any configuration error or something.

neutron.conf (controller node):

[DEFAULT]
core_plugin = ml2
service_plugins = router
allow_overlapping_ips = True
rpc_backend = rabbit
auth_strategy = keystone
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
bind_host = 172.16.102.12

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[database]
connection = mysql+pymysql://neutron:<secret>@controller/neutron

[oslo_concurrency]
lock_path = $state_path/lock

[oslo_messaging_rabbit]
rabbit_hosts=controller01:5672,controller02:5672
rabbit_userid = openstack
rabbit_password = <secret>

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = <secret>

[nova]
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
region_name = K58
project_name = service
username = nova
password = <secret>

plugins/ml2/linuxbridge_agent.ini (controller node):

[linux_bridge]
physical_interface_mappings = public:eth2

[vxlan]
enable_vxlan = True
local_ip = 172.16.102.12
l2_population = True

[agent]
prevent_arp_spoofing = True

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

plugins/ml2/ml2_conf.ini (controller node):

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = linuxbridge,l2population
extension_drivers = port_security

[ml2_type_flat]
flat_networks = *

[ml2_type_vlan]
network_vlan_ranges = public:100:105

[ml2_type_vxlan]
vni_ranges = 1:1000

[securitygroup]
enable_ipset = True

neutron.conf (compute node):

[DEFAULT]
core_plugin = ml2
rpc_backend = rabbit
auth_strategy = keystone
verbose = True
debug = true

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[oslo_concurrency]
lock_path = $state_path/lock

[oslo_messaging_rabbit]
rabbit_hosts=controller01:5672,controller02:5672
rabbit_userid = openstack
rabbit_password = <secret>

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = <secret>

plugins/ml2/linuxbridge_agent.ini (compute node):

[linux_bridge]
physical_interface_mappings = public:eth0.101

[vxlan]
enable_vxlan = True
local_ip = 172.16.102.13
l2_population = True

[agent]
prevent_arp_spoofing = True

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

Any idea, hint, etc?

Many Thanks

Failed to bind port on host

Hi, just made a fresh installation of the Liberty release on Ubuntu 14.04 and I am running into some network problem that I dont understand them. I think the biggest problem is: Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02

Services are running as seen:

neutron agent-list

+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+
| 0a1eb1ea-ec5d-4edd-b2eb-f8adf8545365 | L3 agent           | controller02 | :-)   | True           | neutron-l3-agent          |
| 0cb6076a-135a-42d6-8d73-4bd59f1dbffe | Metadata agent     | controller01 | :-)   | True           | neutron-metadata-agent    |
| 505c0d2d-15f9-4948-9e16-e9a9b0a01d87 | Linux bridge agent | controller02 | :-)   | True           | neutron-linuxbridge-agent |
| 637dc6d3-3c26-42c9-86a3-0246aa357c4f | DHCP agent         | controller02 | :-)   | True           | neutron-dhcp-agent        |
| adadaaa9-bce5-477e-a734-73a3c08f4fb8 | DHCP agent         | controller01 | :-)   | True           | neutron-dhcp-agent        |
| b7985827-ac34-4f38-9245-dc1767c3c008 | Linux bridge agent | hv03         | :-)   | True           | neutron-linuxbridge-agent |
| ef78d68b-20b5-4ea1-804e-f7befa8e2634 | Linux bridge agent | controller01 | :-)   | True           | neutron-linuxbridge-agent |
| f2038985-9f83-47e8-969d-2e91e04e02b7 | Metadata agent     | controller02 | :-)   | True           | neutron-metadata-agent    |
| f82682d9-018d-4cc4-9f94-8302e17be17a | L3 agent           | controller01 | :-)   | True           | neutron-l3-agent          |
| f8f419fc-38d9-470d-925e-fcd45778e32d | Linux bridge agent | hv01         | :-)   | True           | neutron-linuxbridge-agent |
| ff70f7cf-8c74-4642-b14c-32a2a23d91a9 | Linux bridge agent | hv02         | :-)   | True           | neutron-linuxbridge-agent |
+--------------------------------------+--------------------+---------------------+-------+----------------+---------------------------+

neutron net-list

+--------------------------------------+----------+------------------------------------------------------+
| id                                   | name     | subnets                                              |
+--------------------------------------+----------+------------------------------------------------------+
| 78bba7fe-a27d-4000-bd28-704660d9c680 | Internal | d68bb402-da64-42db-897b-51f94e725e3f 172.16.101.0/24 |
+--------------------------------------+----------+------------------------------------------------------+

nova service-list

+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host                | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-cert        | controller01 | internal | enabled | up    | 2016-06-01T15:15:47.000000 | -               |
| 2  | nova-cert        | controller02 | internal | enabled | up    | 2016-06-01T15:15:51.000000 | -               |
| 3  | nova-consoleauth | controller01 | internal | enabled | up    | 2016-06-01T15:15:51.000000 | -               |
| 4  | nova-consoleauth | controller02 | internal | enabled | down  | 2016-06-01T12:38:49.000000 | -               |
| 5  | nova-scheduler   | controller01 | internal | enabled | up    | 2016-06-01T15:15:49.000000 | -               |
| 6  | nova-scheduler   | controller02 | internal | enabled | up    | 2016-06-01T15:15:52.000000 | -               |
| 7  | nova-conductor   | controller01 | internal | enabled | up    | 2016-06-01T15:15:53.000000 | -               |
| 8  | nova-conductor   | controller02 | internal | enabled | up    | 2016-06-01T15:15:49.000000 | -               |
| 9  | nova-compute     | hv01-k58         | nova     | enabled | up    | 2016-06-01T15:15:48.000000 | -               |
| 10 | nova-compute     | hv02-k58         | nova     | enabled | up    | 2016-06-01T15:15:52.000000 | -               |
| 11 | nova-compute     | hv03-k58         | nova     | enabled | up    | 2016-06-01T15:15:50.000000 | -               |
+----+------------------+---------------------+----------+---------+-------+----------------------------+-----------------+

When I start a machine I get following error messages:

neutron-server.log (controller node):

[...]
2016-06-01 16:24:10.164 9575 ERROR neutron.plugins.ml2.managers [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02
2016-06-01 16:24:10.164 9575 ERROR neutron.plugins.ml2.managers [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Failed to bind port 802862a2-223f-4e32-bc9c-b5821351f70b on host hv02
2016-06-01 16:24:10.165 9575 INFO neutron.plugins.ml2.plugin [req-00f88547-b3a3-40bc-9d1e-4bebdefa6fce 859ec55610584042aedd429d341a5242 77cddd1adacb4becb6c0a52188cf34bb - - -] Attempt 2 to bind port 802862a2-223f-4e32-bc9c-b5821351f70b
[...]

nova-conductor.log (controller node):

[...]
2016-06-01 16:24:12.142 1656 ERROR nova.scheduler.utils [req-ec201652-17bc-486c-b5be-7846112d692b d7d8a3b5814540a4b01da3412db9c49c 20f9af47fde64253a5975e072e09ab2e - - -] [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Error from last host: hv02 (node hv02): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2058, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 1a0145bb-55f6-4ea9-9667-9e96876ea17e was re-scheduled: Binding failed for port 802862a2-223f-4e32-bc9c-b5821351f70b, please check neutron logs for more information.\n']
[...]

nova-compute.log (compute node):

[...]
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [req-ec201652-17bc-486c-b5be-7846112d692b d7d8a3b5814540a4b01da3412db9c49c 20f9af47fde64253a5975e072e09ab2e - - -] [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Instance failed to spawn
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] Traceback (most recent call last):
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     yield resources
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     block_device_info=block_device_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2528, in spawn
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     write_to_disk=True)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4401, in _get_guest_xml
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     network_info_str = str(network_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 515, in __str__
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     return self._sync_wrapper(fn, *args, **kwargs)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 498, in _sync_wrapper
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 530, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self[:] = self._gt.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     return self._exit_event.wait()
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     current.throw(*self._exc)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     result = function(*args, **kwargs)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1581, in _allocate_network_async
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     six.reraise(*exc_info)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1564, in _allocate_network_async
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     dhcp_options=dhcp_options)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 744, in allocate_for_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     self._delete_ports(neutron, instance, created_port_ids)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     six.reraise(self.type_, self.value, self.tb)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 736, in allocate_for_instance
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     security_group_ids, available_macs, dhcp_opts)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]   File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 359, in _create_port
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e]     raise exception.PortBindingFailed(port_id=port_id)
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] PortBindingFailed: Binding failed for port 84255ab0-9576-43a5-b175-a75295a4fe60, please check neutron logs for more information.
2016-06-01 16:24:24.577 5889 ERROR nova.compute.manager [instance: 1a0145bb-55f6-4ea9-9667-9e96876ea17e] 
[...]

I dont see any configuration error or something.

neutron.conf (controller node):

[DEFAULT]
core_plugin = ml2
service_plugins = router
allow_overlapping_ips = True
rpc_backend = rabbit
auth_strategy = keystone
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
bind_host = 172.16.102.12

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[database]
connection = mysql+pymysql://neutron:<secret>@controller/neutron

[oslo_concurrency]
lock_path = $state_path/lock

[oslo_messaging_rabbit]
rabbit_hosts=controller01:5672,controller02:5672
rabbit_userid = openstack
rabbit_password = <secret>

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = <secret>

[nova]
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
region_name = K58
project_name = service
username = nova
password = <secret>

plugins/ml2/linuxbridge_agent.ini (controller node):

[linux_bridge]
physical_interface_mappings = public:eth2

[vxlan]
enable_vxlan = True
local_ip = 172.16.102.12
l2_population = True

[agent]
prevent_arp_spoofing = True

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

plugins/ml2/ml2_conf.ini (controller node):

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = linuxbridge,l2population
extension_drivers = port_security

[ml2_type_flat]
flat_networks = *

[ml2_type_vlan]
network_vlan_ranges = public:100:105

[ml2_type_vxlan]
vni_ranges = 1:1000

[securitygroup]
enable_ipset = True

neutron.conf (compute node):

[DEFAULT]
core_plugin = ml2
rpc_backend = rabbit
auth_strategy = keystone
verbose = True
debug = true

[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

[oslo_concurrency]
lock_path = $state_path/lock

[oslo_messaging_rabbit]
rabbit_hosts=controller01:5672,controller02:5672
rabbit_userid = openstack
rabbit_password = <secret>

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = <secret>

plugins/ml2/linuxbridge_agent.ini (compute node):

[linux_bridge]
physical_interface_mappings = public:eth0.101

[vxlan]
enable_vxlan = True
local_ip = 172.16.102.13
l2_population = True

[agent]
prevent_arp_spoofing = True

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

Any idea, hint, etc?etc? One interesting thing is, that no bridge interface was created on the controller nodes. Not sure if this is important at this stage.

Many Thanks