Ask Your Question
0

Ubuntu 12.10 + Folsom + Quantum + OVS + GRE Problems

asked 2012-12-07 17:21:20 -0500

tns9 gravatar image

Hi,

I am having OVS issues with my 3-node (control, network, compute) deployment (Ubuntu 12.10 + Folsom + Quantum + OVS + GRE). From what I can tell, my VM's are given a vnet# interface on the compute node by OVS, but they never are able to reach the network node for DHCP, etc.

I ran a tcpdump last night, which showed a new VM trying repeatedly to get an answer from dnsmasq, which I confirmed is running on 67 udp on the network node.

I'm most concerned by the fact that 'ip netns list' returns nothing on any of the three nodes (running the command as 'root').

It's beginning to look like I should just start again from scratch. I dropped the Quantum db last night and remade it, hoping to pull it all together again, but no matter what I do, nothing seems to be working to fix this issue.

Here is the guide I'm debugging/following. It's based on skible's stable/GRE guide.

https://github.com/josh-wrale/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst (https://github.com/josh-wrale/OpenSta...)

Here's a paste dump of the situation:

http://pastebin.com/raw.php?i=NF34hqMX

Thank you very much for your assistance.

-Joshua

edit retag flag offensive close merge delete

16 answers

Sort by » oldest newest most voted
0

answered 2012-12-07 21:32:50 -0500

tns9 gravatar image

Here is a tcpdump of vnet0 on the compute node, when its only VM is rebooted.

http://pastebin.com/3K1fe5Dd

edit flag offensive delete link more
0

answered 2012-12-09 01:06:29 -0500

gongysh gravatar image

It seems you have not run quantum-openvswitch-agent on your network node right. Do u make sure you are using the same ovs_quantum_plugin.ini file on network node as one on the compute node?

network work node should have br-tun too. We should also have gre port on br-tun on both compute node and network node.

edit flag offensive delete link more
0

answered 2012-12-09 01:07:17 -0500

gongysh gravatar image

Can u give out the: quantum net-show example-net|ext_net?

edit flag offensive delete link more
0

answered 2012-12-07 21:59:39 -0500

tns9 gravatar image

Below, I'm including the full log of the VM booting. Please note that after witnessing the "eth0: IPv6 duplicate address" error in this log, I disabled ipv6 in the sysctl.conf of all three machines and performed a reboot. The log below is from a hard reboot of the VM, after the reboot of the compute node (hypervisor), control node and network node to disable ipv6.

http://pastebin.com/raw.php?i=5PVfUiFw

edit flag offensive delete link more
0

answered 2012-12-09 03:50:19 -0500

tns9 gravatar image

Here are the net-show commands:

http://pastebin.com/raw.php?i=fZw68Rfx

It seems that br-tun should be added automatically, because I configured it in the .ini files. Is br-tun a port which needs to be manually added using ovs-vsctl?

Here is the /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini from the network node:

http://pastebin.com/K41uHUyv

And the compute node:

http://pastebin.com/DJvEhtj4

Thanks, Joshua

edit flag offensive delete link more
0

answered 2012-12-09 07:24:45 -0500

gongysh gravatar image

can u try to run quantum-openvswitch-agent --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini --debug true

on your network node. Of course stop the current one before it.

and paste out the log.

edit flag offensive delete link more
0

answered 2012-12-09 14:03:26 -0500

tns9 gravatar image

Here is the output of quantum-openvswitch-agent:

http://pastebin.com/D22zhiEY

That command appears to have updated the state of OVS:

root@knet-hj29:/etc/init.d# ovs-vsctl show a4de515d-6d78-45b8-870f-abcd97509190 Bridge br-tun Port br-tun Interface br-tun type: internal Port "gre-1" Interface "gre-1" type: gre options: {in_key=flow, out_key=flow, remote_ip="172.20.10.53"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-ex Port "eth2" Interface "eth2" Port br-ex Interface br-ex type: internal Bridge br-int Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal ovs_version: "1.4.3" root@knet-hj29:/etc/init.d#

Here is the agent run on the compute node:

http://pastebin.com/aRhgQeHR

And ovs-vsctl show on the compute node, from after the agent:

root@khyp-c49x:/etc/init.d# service quantum-plugin-openvswitch-agent startquantum-plugin-openvswitch-agent start/running, process 9967 root@khyp-c49x:/etc/init.d# service quantum-plugin-openvswitch-agent stopstop: Unknown instance: root@khyp-c49x:/etc/init.d# service quantum-plugin-openvswitch-agent start quantum-plugin-openvswitch-agent start/running, process 10096 root@khyp-c49x:/etc/init.d# ovs-vsctl show 801aa35e-5de6-483d-a692-6555ea348f87 Bridge br-int Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port br-int Interface br-int type: internal Port "qvoe4bc93cc-d5" tag: 1 Interface "qvoe4bc93cc-d5" Bridge br-tun Port "gre-2" Interface "gre-2" type: gre options: {in_key=flow, out_key=flow, remote_ip="172.20.10.52"} Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} ovs_version: "1.4.3" root@khyp-c49x:/etc/init.d#

I just tested another VM. It is still unable to reach DCHP. It seems like something is making the agents crash. Is the following behavior normal? It seems strange to see the agents on both compute and network exit'ing with code 1 in dmesg.

(tail of dmesg on compute node) [36275.758262] block nbd15: queue cleared [36277.235971] type=1400 audit(1355061400.942:23): apparmor="STATUS" operation="profile_load" name="libvirt-3c597e88-1a5a-4147-8217-43f447c240d2" pid=10822 comm="apparmor_parser" [36277.349062] device vnet0 entered promiscuous mode [36277.355463] qbrfe616ec2-11: port 2(vnet0) entered forwarding state [36277.355488] qbrfe616ec2-11: port 2(vnet0) entered forwarding state [36278.958207] kvm: 10992: cpu0 unhandled rdmsr: 0xc0010112 [36289.833629] qbrfe616ec2-11: port 1(qvbfe616ec2-11) entered forwarding state [36292.390485] qbrfe616ec2-11: port 2(vnet0) entered forwarding state root@khyp-c49x:/etc/init.d# service quantum-plugin-openvswitch-agent stop stop: Unknown instance: root@khyp-c49x:/etc/init.d# service quantum-plugin-openvswitch-agent start quantum-plugin-openvswitch-agent start/running, process 11341 root@khyp-c49x:/etc/init.d# dmesg|tail [36275.757391] block nbd15: Unexpected reply (ffff883fd06c5c48) [36275.758262] block nbd15: queue cleared [36277.235971] type=1400 audit(1355061400.942:23): apparmor="STATUS" operation="profile_load" name="libvirt-3c597e88-1a5a-4147-8217-43f447c240d2" pid=10822 comm="apparmor_parser" [36277.349062] device vnet0 entered promiscuous mode [36277.355463] qbrfe616ec2-11: port 2(vnet0) entered forwarding state [36277.355488] qbrfe616ec2-11: port 2(vnet0) entered forwarding state [36278.958207] kvm: 10992: cpu0 unhandled rdmsr: 0xc0010112 [36289.833629] qbrfe616ec2-11: port 1(qvbfe616ec2-11) entered forwarding state [36292.390485] qbrfe616ec2-11: port 2(vnet0) entered forwarding state [36407.439814] init: quantum-plugin-openvswitch-agent main process (11341) terminated with status 1 root@khyp-c49x:/etc/init.d#

And also this (on network node):

root@knet-hj29:/etc/init.d# dmesg|tail [ 129 ... (more)

edit flag offensive delete link more
0

answered 2012-12-09 15:11:06 -0500

gongysh gravatar image

try to run agents on both network and compute nodes with direct command line. It seems your dhcp agent is not running, either. You should start it on your network node too: sudo quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/dhcp_agent.ini --debug true

By the way, I don't know why your service way to start the agents do not work.

edit flag offensive delete link more
0

answered 2012-12-09 16:18:57 -0500

tns9 gravatar image

Your hint led me to resolve a major issue: In trying to debug the network and compute nodes, I purged OVS/Quantum, backed up /var/log and 'rm -rf /var/log/*'. Well, that was a stupid idea, because the /var/log/quantum and /var/log/upstart directories are not automatically spawned if not present. This behavior was keeping the quantum services from starting. I recreated them, per the control node's example, including permissions. All of the agents appear to be started now.

However, my VM is still not able to get a DHCP address.

The following tcpdump comes from the network node, where the l3-agent and dhcp-agent live. It confirms that the DHCP request is arriving on the 'tap' of the network node.

http://pastebin.com/xbUVtVjR

I'm now working on the next layer of the onion. :-)

Thanks, Joshua

edit flag offensive delete link more
0

answered 2012-12-10 00:21:09 -0500

gongysh gravatar image

Don't forget your network node also need run l2 agent quantum-openvswitch-agent. and please 'ovs-vsctl show' on both network and compute node when u are certain all agents run well.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-12-07 17:21:20 -0500

Seen: 73 times

Last updated: Feb 19 '13