Ask Your Question
0

Neutron router hosted on rebuilt controller fails to route as expected.

asked 2015-08-20 18:01:55 -0500

bnordgren gravatar image

I have a 6 node (5 compute/storage; 1 controller) RDO Kilo openstack cluster installed using a packstack answer file. The cluster behaved as expected until power supply on controller failed. I put another machine in place as a controller, using the same CentOS 7 base OS and the same packstack answer file. The mysql database from the failed controller has been restored from backups. Glance images were NFS mounted, and cinder storage is out on the compute nodes.

Everything except neutron appears to be working. I can start and stop instances, log in to instances via the web console, etc. Neutron shows my old security groups and the instances have the correct security groups applied. The neutron agents appear to all be running (see below). The problem is that my VMs can't access the outside world (ping and ssh), and the outside world can't access the VMs (ping, ssh, http).

[root@plume recovery(keystone_admin)]# neutron agent-list
+--------------------+-----------------------+-------+----------------+---------------------------+
| agent_type         | host                  | alive | admin_state_up | binary                    |
+--------------------+-----------------------+-------+----------------+---------------------------+
| Open vSwitch agent | compute-0-0.plume     | :-)   | True           | neutron-openvswitch-agent |
| Open vSwitch agent | plume.usfs-i2.umt.edu | :-)   | True           | neutron-openvswitch-agent |
| Open vSwitch agent | compute-0-3.plume     | :-)   | True           | neutron-openvswitch-agent |
| Open vSwitch agent | compute-0-4.plume     | :-)   | True           | neutron-openvswitch-agent |
| Open vSwitch agent | compute-0-1.plume     | :-)   | True           | neutron-openvswitch-agent |
| L3 agent           | plume.usfs-i2.umt.edu | :-)   | True           | neutron-l3-agent          |
| DHCP agent         | plume.usfs-i2.umt.edu | :-)   | True           | neutron-dhcp-agent        |
| Metadata agent     | plume.usfs-i2.umt.edu | :-)   | True           | neutron-metadata-agent    |
| Open vSwitch agent | compute-0-2.plume     | :-)   | True           | neutron-openvswitch-agent |
+--------------------+-----------------------+-------+----------------+---------------------------+

Using the suggested tcpdump command from the Openstack Operations manual on the controller node, which is the only node having a connection to the external network, I get the following results as I ping from various places.

The tcpdump command is:

tcpdump -i any -n -v \ 'icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-echo'
  • Ping from external machine to VM: "Destination host unreachable"; no output from tcpdump
  • Ping from external machine to controller node: timely ping replies; tcpdump logs every ping and reply
  • Ping from VM to router's internal interface: 18 ping replies; tcpdump logs only the first request, not the reply and not subsequent requests
  • Ping from VM to external machine: "Destination host unreachable"; no output from tcpdump
  • Ping from VM to router's external interface: timely ping replies; no output from tcpdump
  • Ping from external machine to router's external interface: "Destination host unreachable"; no output from tcpdump
  • Ping from controller node to router's external interface: "Destination host unreachable"; no output from tcpdump

Can anyone help me figure out why neutron isn't routing as it should? Was it maintaining some state somewhere other than the database and I just didn't transfer it to the replacement controller?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-08-21 16:47:06 -0500

bnordgren gravatar image

Captain genius forgot to update the external network interface name in the answer file when he swapped out machines. The br-ex bridge had interface "enp2s0", which doesn't exist, instead of correctly listing "eno1". I'm not sure why this isn't an error.

Corrected the information in the packstack answer file, re-ran packstack, restarted all the neutron services on the controller and viola. Problem solved.

edit flag offensive delete link more

Comments

Looks like you learnt a bit while trying to find your typo! nothing ventured, nothing gained!

lukehinds gravatar imagelukehinds ( 2015-08-21 17:43:55 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-08-20 18:01:55 -0500

Seen: 328 times

Last updated: Aug 21 '15