malformed br-tun - where to look for answers?

Context:

• Juno setup according to http://docs.openstack.org/juno/instal...
• Four virtual machines as physical hosts (controller, network, two compute), running Centos 7
• Neutron with GRE tunnels to implement tenant networks

Problem: Instances don't get any reply from the DHCP server. It turns out that the br-tun bridge on the network node is missing its GRE port:

# ovs-vsctl list-ports br-tun
patch-int
# ovs-vsctl show
f5eb5f84-a5ca-4747-b745-a35b1a997eab
Bridge br-int
(...)
Bridge br-tun
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal


GRE-encapsulated DHCP requests do arrive at the network node, but they are then rejected with an ICMP "admin prohibited". Not surprising given the lack of a full tunnel.

I don't see any relevant error in the neutron log files, neither on the controller nor the network node. My config files seem correct, but earlier a number of things were misconfigured and I wonder whether the database contains incorrect data as a result.

Since this is a test installation, I can reinstall everything, but this is time-consuming and a bit unsatisfactory. Where do I start figuring out what went wrong, and fix it?

edit retag close merge delete

admin prohibited normally means icmp is blocked either using iptables or echo is disabled

( 2014-12-09 23:06:29 -0500 )edit

( 2014-12-10 00:02:52 -0500 )edit

@anantha: When using Centos and RedHat, the firewall is indeed an important factor. Not in my case though. Here, "admin prohibited" means probably "I don't know what to do with this GRE packet".

( 2014-12-10 00:12:58 -0500 )edit

@dbaxps: Thanks much, Boris. My problem is solved now, and the offending line was indeed in the ml2_conf.ini file, namely:

[ovs]


I copied and pasted from the installation instructions and forgot to enter the correct address.

( 2014-12-10 00:15:53 -0500 )edit

@dbaxps: To be precise, it's the ml2_conf.ini files on the compute nodes. I had checked the config on the controller and network nodes and not found anything wrong.

( 2014-12-10 00:17:56 -0500 )edit

Sort by » oldest newest most voted

Problem solved. I was wrong on two accounts.

There was a clue in the log files: On the network node, /var/log/neutron/openvswitch-agent.log contains a WARNING "Unable to create tunnel port. Invalid remote IP: INSTANCE_TUNNELS_INTERFACE_IP_ADDRESS". At first, I only looked for ERRORs.

My configuration was indeed wrong: I had copied and pasted the instructions and not adapted them to my IP address. As a result, the ml2_conf.ini files on the compute nodes contained the line

local_ip=INSTANCE_TUNNELS_INTERFACE_IP_ADDRESS


This can also be seen in the database. The command mysqldump --all-databases -u root -p --result-file=/tmp/t writes the whole DB out to /tmp/t; the incorrect IP address is, among other locations, in a table named agents. The wrong address can also be seen in the output of neutron agent-show ID_of_Openvswitch_agent_on_compute_node.

After correcting the addresses and restarting the neutron-openvswitch-agent service on the compute nodes, the GRE ports appeared in the br-tun bridge. The networking problems have gone away.

Major takeaways:

1. config files on compute nodes impact the OVS setup on the network node
2. WARNINGs in log files can be relevant
more

delete the bridge and recreate

more

Would you recommend recreating it manually? This would require more insight than I have. In any case, my problem was due to bad configuration on the compute nodes; had I just deleted the bridge and restarted the neutron processes, the same malformed bridge would have been created.

( 2014-12-10 00:43:49 -0500 )edit