No network communication on a 3 node Havana setup
asked 2013-12-23 11:16:24 -0600
I've been struggling with neutron for the past few days and I can't get instances to receive addresses using DHCP or reach anything outside the same physical host. My setup consists of three nodes, all running Ubuntu Server 12.04 with Havana installed from Cloudarchive (per http://docs.openstack.org/trunk/install-guide/install/apt/content/ ):
- Network node: Connected to the WAN (eth0) and LAN (eth1, 10.130.10.201) networks running neutron dhcp-agent, l3-agent, metadata-agent and plugin-openvswitch-agent.
- Management node: Connected only to the LAN network (eth0, 10.130.10.202) and running neutron (server, plugin-openvswitch-agent), postgresql, rabbitmq, keystone, glance, nova (api, conductor, cert, vnc proxy and scheduler) and horizon.
- Compute node: Connected only to the LAN network (eth0, 10.130.10.11) and running nova-compute and neutron-plugin-openvswitch-agent
Everything aside from networking works as expected so far, and two instances running on the same compute node are able to reach each other if I log on using the console and assign an address manually.
On the neutron server log (management node) I can see the following message whenever I launch an instance:
2013-12-23 12:52:13.151 862 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'be9659ca-1ee7-4c35-b36c-d082a581495f'], 'name': u'Alpha', 'provider:physical_network': None, 'admin_state_up': True, 'tenant_id': u'04968d54151d4bb29d477f754e099728', 'provider:network_type': u'local', 'router:external': False, 'shared': False, 'id': u'031dcf57-c259-41f2-be2a-1541f88d3238', 'provider:segmentation_id': None}
General info:
- neutron net list: http://pastebin.com/J9Y7w9iS
- neutron subnets: http://pastebin.com/pqFyVQPy
- neutron routers: http://pastebin.com/A8mf7gUz
Configuration files and logs follow:
Network node:
- /etc/neutron/neutron.conf: http://pastebin.com/3VY2GcPA
- /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini: http://pastebin.com/0CpVyiZs
- /etc/neutron/l3_agent.ini: http://pastebin.com/kLCUd6HW
- /etc/neutron/api-paste.ini: http://pastebin.com/QU7MwsB8
- /etc/neutron/dhcp_agent.ini: http://pastebin.com/ReHsfYUE
- /etc/neutron/dhcp_agent.ini: metadata_agent.ini http://pastebin.com/tX6bhxkD
- ovs-vsctl show: http://pastebin.com/HDtCrnFH
- ovs-ofctl show br-tun: http://pastebin.com/87SShbjB
Compute node:
- /etc/nova/nova.conf: http://pastebin.com/eR4czuEM
- /etc/neutron/neutron.conf: http://pastebin.com/4GNZyyjc
- /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini: http://pastebin.com/yA9XiJGA
- ovs-vsctl show: http://pastebin.com/YG0SpmLQ
- ovs-ofctl show br-tun: http://pastebin.com/1YNAj0qH
Management Node:
- Server Log: http://pastebin.com/AdKCfUQh
- /etc/nova/nova.conf: http://pastebin.com/3kDt53zY
- /etc/neutron/neutron.conf: http://pastebin.com/5DSddTsu
- /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini: http://pastebin.com/G11HbZER
Tcpdump output when launching a new instance:
- Compute node: http://pastebin.com/A8Gkst4W
- Network node: http://pastebin.com/XTAKA7fA
That error is innocuous most likely (I get the error). See https://bugs.launchpad.net/neutron/+bug/1192786 Can you post your nova.conf file?
SamYaple ( 2013-12-23 13:43:44 -0600 )editI've added the nova.conf files to the description (both on the compute and management nodes)
Diego Lima ( 2013-12-23 14:12:01 -0600 )editYou are missing some information in your neutron.conf file under the [DEFAULT] header. You need the keystone auth stuff under that (though I am unsure why, I ran into the same issue). See http://pastebin.com/A0ZS33rD Also, your neutron.conf files conflict (allownamespace), use one file.
SamYaple ( 2013-12-23 14:31:51 -0600 )editI've made the suggested changes but still get no networking. Upon launching my vm I can see some traffic (dhcp requests) on the compute node's br-int interface, but no traffic on the network node. I'll edit my main post and place the tcpdump results here. I've also updated pastebin's conf files.
Diego Lima ( 2013-12-23 15:26:34 -0600 )editWell you are on the right track. This is a bridging issue now. I still see an entry for br-ex on your network node, have you removed that one? Apply changes from http://pastebin.com/A0ZS33rD Once you remove all br-ex from all nodes, retry.
SamYaple ( 2013-12-23 15:37:57 -0600 )edit