Revision history [back]

click to hide/show revision 1
initial version

Failed to bind port on host neutron.plugins.ml2.managers (SLES 11SP3, ML2/VLAN, running under Hyper-V

I'm having an issue that had me pulling my hair out. I've been able to get everything else working fine with OpenStack icehouse on SLES except the Networking. I've searched the various forums, looked at parts of the code to no avail...

In summary, I cannot get networking to config properly on the dedicated network node. (See errors at bottom)

I have three adapters on the node:

eth0 -> public interface 10.36.192.64/26 gw 10.36.192.65 (VLAN Trunked in Hyper-V 0-99 with default vlan0) and set to promiscuous mode

eth1 -> management interface 10.5.5.0/24 (no vlan)

eth2 -> data/vm network (VLAN trunked in Hyper-V 100-200 with default vlan 100) testing with VLAN 100 with subnet 10.5.6.0/24

br-ex created with eth0 as the added port (has active IP address) br-eth2 created with eth2 as the added port (No IP Address)

Public interface into the server works fine as does the management interface.

/usr/bin/ovs-vsctl show
54a37a2e-428b-4ed4-8de5-e97c7c763de2
    Bridge "br-eth2"
        Port "br-eth2"
            Interface "br-eth2"
                type: internal
        Port "eth2"
            Interface "eth2"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth0"
            Interface "eth0"
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "qr-fc50245e-f8"
            tag: 4095
            Interface "qr-fc50245e-f8"
                type: internal
        Port "qg-5519f1d6-d2"
            tag: 4095
            Interface "qg-5519f1d6-d2"
                type: internal
    ovs_version: "2.0.0"

Here is the net/subnet/router definitions:

controller01:/var/log/neutron # neutron net-list
+--------------------------------------+----------+------------------------------------------------------+
| id                                   | name     | subnets                                              |
+--------------------------------------+----------+------------------------------------------------------+
| 44d7053e-e845-4fe6-9412-a61133638670 | demo-net | 588638e7-b14a-4612-b3df-d517ff45533e 10.5.6.0/24     |
| 9779e373-e865-443a-a0dc-c7ea8f19fa20 | ext-net  | 75f978ce-2733-4b66-8f5a-a96f17e5f87a 10.36.192.64/26 |
+--------------------------------------+----------+------------------------------------------------------+
controller01:/var/log/neutron # neutron net-show 9779e373-e865-443a-a0dc-c7ea8f19fa20
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 9779e373-e865-443a-a0dc-c7ea8f19fa20 |
| name                      | ext-net                              |
| provider:network_type     | vlan                                 |
| provider:physical_network | extnet                               |
| provider:segmentation_id  | 2                                    |
| router:external           | True                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 75f978ce-2733-4b66-8f5a-a96f17e5f87a |
| tenant_id                 | 6f3375bde62643d5bfbbd241e58976d9     |
+---------------------------+--------------------------------------+
controller01:/var/log/neutron # neutron net-show 44d7053e-e845-4fe6-9412-a61133638670
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 44d7053e-e845-4fe6-9412-a61133638670 |
| name                      | demo-net                             |
| provider:network_type     | vlan                                 |
| provider:physical_network | datanet                              |
| provider:segmentation_id  | 100                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 588638e7-b14a-4612-b3df-d517ff45533e |
| tenant_id                 | 238f9e1e2c4346f8a233a320d04c7794     |
+---------------------------+--------------------------------------+
controller01:/var/log/neutron # neutron subnet-list
+--------------------------------------+------+-----------------+----------------------------------------------------+
| id                                   | name | cidr            | allocation_pools                                   |
+--------------------------------------+------+-----------------+----------------------------------------------------+
| 588638e7-b14a-4612-b3df-d517ff45533e |      | 10.5.6.0/24     | {"start": "10.5.6.2", "end": "10.5.6.254"}         |
| 75f978ce-2733-4b66-8f5a-a96f17e5f87a |      | 10.36.192.64/26 | {"start": "10.36.192.115", "end": "10.36.192.124"} |
+--------------------------------------+------+-----------------+----------------------------------------------------+
controller01:/var/log/neutron # neutron subnet-show 588638e7-b14a-4612-b3df-d517ff45533e
+------------------+--------------------------------------------+
| Field            | Value                                      |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.5.6.2", "end": "10.5.6.254"} |
| cidr             | 10.5.6.0/24                                |
| dns_nameservers  |                                            |
| enable_dhcp      | True                                       |
| gateway_ip       | 10.5.6.1                                   |
| host_routes      |                                            |
| id               | 588638e7-b14a-4612-b3df-d517ff45533e       |
| ip_version       | 4                                          |
| name             |                                            |
| network_id       | 44d7053e-e845-4fe6-9412-a61133638670       |
| tenant_id        | 238f9e1e2c4346f8a233a320d04c7794           |
+------------------+--------------------------------------------+
controller01:/var/log/neutron # neutron subnet-show 75f978ce-2733-4b66-8f5a-a96f17e5f87a
+------------------+----------------------------------------------------+
| Field            | Value                                              |
+------------------+----------------------------------------------------+
| allocation_pools | {"start": "10.36.192.115", "end": "10.36.192.124"} |
| cidr             | 10.36.192.64/26                                    |
| dns_nameservers  |                                                    |
| enable_dhcp      | False                                              |
| gateway_ip       | 10.36.192.65                                       |
| host_routes      |                                                    |
| id               | 75f978ce-2733-4b66-8f5a-a96f17e5f87a               |
| ip_version       | 4                                                  |
| name             |                                                    |
| network_id       | 9779e373-e865-443a-a0dc-c7ea8f19fa20               |
| tenant_id        | 6f3375bde62643d5bfbbd241e58976d9                   |
+------------------+----------------------------------------------------+
controller01:/var/log/neutron # neutron router-list
+--------------------------------------+------------+-----------------------------------------------------------------------------+
| id                                   | name       | external_gateway_info                                                       |
+--------------------------------------+------------+-----------------------------------------------------------------------------+
| aa8db70a-25dc-4fed-8814-ea79bf5bf404 | ext-to-int | {"network_id": "9779e373-e865-443a-a0dc-c7ea8f19fa20", "enable_snat": true} |
+--------------------------------------+------------+-----------------------------------------------------------------------------+
controller01:/var/log/neutron # neutron router-show aa8db70a-25dc-4fed-8814-ea79bf5bf404
+-----------------------+-----------------------------------------------------------------------------+
| Field                 | Value                                                                       |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up        | True                                                                        |
| external_gateway_info | {"network_id": "9779e373-e865-443a-a0dc-c7ea8f19fa20", "enable_snat": true} |
| id                    | aa8db70a-25dc-4fed-8814-ea79bf5bf404                                        |
| name                  | ext-to-int                                                                  |
| routes                |                                                                             |
| status                | ACTIVE                                                                      |
| tenant_id             | 238f9e1e2c4346f8a233a320d04c7794                                            |
+-----------------------+-----------------------------------------------------------------------------+

Here are the config files from the controller and network node (seperated by =====)

=====================================================================
controller01:/etc/neutron # grep -v -e "^#" -e "^$" neutron.conf
[DEFAULT]
verbose = True
debug = True
lock_path = /var/run/neutron
control_exchange = neutron
notification_driver = neutron.openstack.common.notifier.rpc_notifier
verbose = True
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
state_path = /var/lib/neutron
log_dir = /var/log/neutron
auth_strategy = keystone
rpc_backend = neutron.openstack.common.rpc.impl_kombu
rabbit_host = controller
rabbit_userid = guest
rabbit_password = xxxx
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
nova_admin_username = nova
nova_admin_tenant_id =
nova_admin_password = xxxx
nova_admin_auth_url = http://controller:35357/v2.0
[quotas]
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = neutron
admin_password = xxxx
signing_dir = /var/cache/neutron/keystone-signing
auth_uri = http://controller:5000
[database]
connection = mysql://neutron:xxxx@controller/neutron
[service_providers]
service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
service_provider=VPN:openswan:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default

=================================================================================
controller01:/etc/neutron/plugins/ml2 # grep -v -e "^#" -e "^$" ml2_conf.ini
[ml2]
mechanism_drivers = openvswitch
type_drivers = vlan
tenant_network_types = vlan
[ml2_type_flat]
[ml2_type_vlan]
network_vlan_ranges = extnet:1:99,datanet:100:200
[ml2_type_gre]
[ml2_type_vxlan]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
[ovs]
bridge_mappings = extnet:br-ex,datanet:br-eth2

====================================================================================
netnode01:/etc/neutron #  grep -v -e "^#" -e "^$" neutron.conf
[DEFAULT]
verbose = True
lock_path = /var/run/neutron
control_exchange = neutron
notification_driver = neutron.openstack.common.notifier.rpc_notifier
verbose = True
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin
state_path = /var/lib/neutron
log_dir = /var/log/neutron
auth_strategy = keystone
rpc_backend = neutron.openstack.common.rpc.impl_kombu
rabbit_host = controller
rabbit_userid = guest
rabbit_password = xxxx
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
nova_admin_username = nova
nova_admin_tenant_id =
nova_admin_password = xxxx
nova_admin_auth_url = http://controller:35357/v2.0
[quotas]
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = neutron
admin_password = xxxx
signing_dir = /var/cache/neutron/keystone-signing
auth_uri = http://controller:5000
[database]
connection = mysql://neutron:xxxx@controller/neutron
[service_providers]
service_provider=LOADBALANCER:Haproxy:neutron.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
service_provider=VPN:openswan:neutron.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default

=================================================================================================================================
netnode01:/etc/neutron #  grep -v -e "^#" -e "^$" l3_agent.ini
[DEFAULT]
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
use_namespaces = True
debug = True
external_network_bridge =
gateway_external_network_id =

====================================================================================
netnode01:/etc/neutron/plugins/ml2 # grep -v -e "^#" -e "^$" ml2_conf.ini
[ml2]
mechanism_drivers = openvswitch
type_drivers = vlan
tenant_network_types = vlan
[ml2_type_flat]
[ml2_type_vlan]
network_vlan_ranges = extnet:1:99,datanet:100:200
[ml2_type_gre]
[ml2_type_vxlan]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
[ovs]
bridge_mappings = extnet:br-ex,datanet:br-eth2

and the namespace information:

netnode01:/etc/neutron/plugins/ml2 # ip netns
qrouter-aa8db70a-25dc-4fed-8814-ea79bf5bf404
netnode01:/etc/neutron/plugins/ml2 # ip netns exec qrouter-aa8db70a-25dc-4fed-8814-ea79bf5bf404 ip a
9: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
10: qg-5519f1d6-d2: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:4c:9f:07 brd ff:ff:ff:ff:ff:ff
    inet 10.36.192.115/26 brd 10.36.192.127 scope global qg-5519f1d6-d2
11: qr-fc50245e-f8: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:bc:0d:40 brd ff:ff:ff:ff:ff:ff
    inet 10.5.6.1/24 brd 10.5.6.255 scope global qr-fc50245e-f8
netnode01:/etc/neutron/plugins/ml2 #

Note that I cannot ping the external gateway from qg-5519f1d6-d2

netnode01:/etc/neutron/plugins/ml2 # ip netns exec qrouter-aa8db70a-25dc-4fed-8814-ea79bf5bf404 ping 10.36.192.65
PING 10.36.192.65 (10.36.192.65) 56(84) bytes of data.
From 10.36.192.115: icmp_seq=2 Destination Host Unreachable
From 10.36.192.115 icmp_seq=2 Destination Host Unreachable
From 10.36.192.115 icmp_seq=3 Destination Host Unreachable
From 10.36.192.115 icmp_seq=4 Destination Host Unreachable

And here are the errors I am seeing:

2014-08-13 18:24:23.178 6875 DEBUG neutron.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_msg_id': u'd46d029f72c442ba963a8e9ccd505e97', u'args': {u'host': u'netnode01', u'router_ids': None}, u'_context_read_deleted': u'no', u'_context_user_name': None, u'_context_request_id': u'req-74015492-962a-4851-9978-35f0b230b494', u'_reply_q': u'reply_1e36fe35e3e34267bd5518e4bee9108d', u'_context_tenant_id': None, u'_context_tenant_name': None, u'_context_tenant': None, u'_unique_id': u'd30e19a82e8846f5a0f724b4dff11f1f', u'_context_project_name': None, u'_context_is_admin': True, u'version': u'1.0', u'_context_project_id': None, u'_context_timestamp': u'2014-08-13 22:24:23.168753', u'_context_user': None, u'_context_user_id': None, u'namespace': None, u'method': u'sync_routers'} _safe_log /usr/lib64/python2.6/site-packages/neutron/openstack/common/rpc/common.py:280

2014-08-13 18:24:23.179 6875 DEBUG neutron.openstack.common.rpc.amqp [-] unpacked context: {'project_name': None, 'user_id': None, 'roles': [u'admin'], 'tenant_id': None, 'timestamp': u'2014-08-13 22:24:23.168753', 'is_admin': True, 'read_deleted': u'no', 'request_id': u'req-74015492-962a-4851-9978-35f0b230b494', 'tenant_name': None, 'project_id': None, 'user_name': None, 'tenant': None, 'user': None} _safe_log /usr/lib64/python2.6/site-packages/neutron/openstack/common/rpc/common.py:280

2014-08-13 18:24:23.180 6875 DEBUG neutron.context [req-74015492-962a-4851-9978-35f0b230b494 None] Arguments dropped when creating context: {'project_name': None, 'tenant': None} __init__ /usr/lib64/python2.6/site-packages/neutron/context.py:84

2014-08-13 18:24:23.183 6875 DEBUG neutron.common.utils [req-74015492-962a-4851-9978-35f0b230b494 None] Reloading cached file /etc/neutron/policy.json read_cached_file /usr/lib64/python2.6/site-packages/neutron/common/utils.py:112

2014-08-13 18:24:23.184 6875 DEBUG neutron.policy [req-74015492-962a-4851-9978-35f0b230b494 None] Loading policies from file: /etc/neutron/policy.json _set_rules /usr/lib64/python2.6/site-packages/neutron/policy.py:89

2014-08-13 18:24:23.200 6875 DEBUG neutron.scheduler.l3_agent_scheduler [req-74015492-962a-4851-9978-35f0b230b494 None] No non-hosted routers auto_schedule_routers /usr/lib64/python2.6/site-packages/neutron/scheduler/l3_agent_scheduler.py:97

2014-08-13 18:24:23.319 6875 DEBUG neutron.db.l3_rpc_base [req-74015492-962a-4851-9978-35f0b230b494 None] Checking router: b535ef8b-f24f-42ac-a62e-1e06f50f694a for host: netnode01 _ensure_host_set_on_ports /usr/lib64/python2.6/site-packages/neutron/db/l3_rpc_base.py:71

2014-08-13 18:24:23.364 6875 DEBUG neutron.plugins.ml2.managers [req-74015492-962a-4851-9978-35f0b230b494 None] Attempting to bind port fc102c37-2049-4493-818e-c495ea497e34 on host netnode01 for vnic_type normal with profile {} bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/managers.py:445

2014-08-13 18:24:23.365 6875 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-74015492-962a-4851-9978-35f0b230b494 None] Attempting to bind port fc102c37-2049-4493-818e-c495ea497e34 on network 3dabd4c7-a0f0-4540-952c-ffeadebbfecf bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:58

2014-08-13 18:24:23.371 6875 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-74015492-962a-4851-9978-35f0b230b494 None] Checking agent: {'binary': u'neutron-openvswitch-agent', 'description': None, 'admin_state_up': True, 'heartbeat_timestamp': datetime.datetime(2014, 8, 13, 22, 24, 20), 'alive': True, 'id': u'e8f16ad0-6a0d-46ea-b054-fd0e8e84e59c', 'topic': u'N/A', 'host': u'netnode01', 'agent_type': u'Open vSwitch agent', 'started_at': datetime.datetime(2014, 8, 13, 22, 24, 20), 'created_at': datetime.datetime(2014, 8, 13, 18, 9, 51), 'configurations': {u'tunnel_types': [], u'tunneling_ip': u'', u'bridge_mappings': {}, u'l2_population': False, u'devices': 0}} bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:66

2014-08-13 18:24:23.372 6875 DEBUG neutron.plugins.ml2.drivers.mech_openvswitch [req-74015492-962a-4851-9978-35f0b230b494 None] Checking segment: {'segmentation_id': 2L, 'physical_network': u'extnet', 'id': u'2795628b-4fd8-44da-abc6-a0e4967e22fe', 'network_type': u'vlan'} for mappings: {} with tunnel_types: [] check_segment_for_agent /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_openvswitch.py:49

2014-08-13 18:24:23.373 6875 WARNING neutron.plugins.ml2.managers [req-74015492-962a-4851-9978-35f0b230b494 None] Failed to bind port fc102c37-2049-4493-818e-c495ea497e34 on host netnode01
2014-08-13 18:24:23.516 6875 DEBUG neutron.plugins.ml2.managers [req-74015492-962a-4851-9978-35f0b230b494 None] Attempting to bind port d283cc10-4433-4324-923d-072d8b6145fd on host netnode01 for vnic_type normal with profile {} bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/managers.py:445

2014-08-13 18:24:23.517 6875 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-74015492-962a-4851-9978-35f0b230b494 None] Attempting to bind port d283cc10-4433-4324-923d-072d8b6145fd on network b7f3c703-a290-40c3-ba06-b5630a5cd94c bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:58

2014-08-13 18:24:23.523 6875 DEBUG neutron.plugins.ml2.drivers.mech_agent [req-74015492-962a-4851-9978-35f0b230b494 None] Checking agent: {'binary': u'neutron-openvswitch-agent', 'description': None, 'admin_state_up': True, 'heartbeat_timestamp': datetime.datetime(2014, 8, 13, 22, 24, 20), 'alive': True, 'id': u'e8f16ad0-6a0d-46ea-b054-fd0e8e84e59c', 'topic': u'N/A', 'host': u'netnode01', 'agent_type': u'Open vSwitch agent', 'started_at': datetime.datetime(2014, 8, 13, 22, 24, 20), 'created_at': datetime.datetime(2014, 8, 13, 18, 9, 51), 'configurations': {u'tunnel_types': [], u'tunneling_ip': u'', u'bridge_mappings': {}, u'l2_population': False, u'devices': 0}} bind_port /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_agent.py:66

2014-08-13 18:24:23.525 6875 DEBUG neutron.plugins.ml2.drivers.mech_openvswitch [req-74015492-962a-4851-9978-35f0b230b494 None] Checking segment: {'segmentation_id': 1000L, 'physical_network': u'datanet', 'id': u'263a641b-4f90-4bff-a931-12f8d6974d63', 'network_type': u'vlan'} for mappings: {} with tunnel_types: [] check_segment_for_agent /usr/lib64/python2.6/site-packages/neutron/plugins/ml2/drivers/mech_openvswitch.py:49

2014-08-13 18:24:23.526 6875 WARNING neutron.plugins.ml2.managers [req-74015492-962a-4851-9978-35f0b230b494 None] Failed to bind port d283cc10-4433-4324-923d-072d8b6145fd on host netnode01

2014-08-13 18:24:23.573 6875 DEBUG neutron.db.l3_rpc_base [req-74015492-962a-4851-9978-35f0b230b494 None] Routers returned to l3 agent:

netnode01:/var/log/neutron # tail -f neutron-openvswitch-agent.log | grep WARNING 2014-08-13 20:32:41.947 3833 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-850375ee-1a55-4f37-af42-090947bcf11c None] Device 5519f1d6-d2ee-4f73-803b-f92b468ea66d not defined on plugin

2014-08-13 20:33:50.044 3833 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-850375ee-1a55-4f37-af42-090947bcf11c None] Device fc50245e-f8d4-4dbf-be78-9d3b6cbb754c not defined on plugin

As I said earlier