Ask Your Question
3

Changes made to Neutron not being implemented [closed]

asked 2014-03-13 16:39:50 -0500

zxc123 gravatar image

updated 2014-03-26 23:12:21 -0500

Hi,

I have been running a havana openstack cloud with neutron networking for 2 months now without any issues but recently I have run into a problem where any changes made to the neutron system don't seem to be implemented.

When I create a new instance or assign a floating IP address there are no error messages, the instance is assigned an IP address and everything looks normal on the dashboard, but the instance's network card cannot connect to the network and traffic cannot pass through new floating IP addresses. Exisiting machines and floating IP addresses are not affected by this problem and everything that was already setup carries on working.

It is possible to fix the problem by rebooting all the servers, but that is not a good solution as it will start happening again a week later.

I cannot find any errors for this in any of the neutron logs, though I have found that on openvswitch port isn't being created for new instances.

Does anyone know why this would be happening or what steps I can take to diognose the probem?


Edit: it happened again today, I cannot find anything relating to networking in the nova-compute logs or libvirt logs. I think libvirt is working as the machine's network card shows the correct mac address - traffic just never reaches any other machines.
After creating a new machine, Neutron does show that a port was created but with a status of DOWN . I found the instance in libvirt and used virsh dumpxml instance-0000005b to get <target dev='tapa65c9a88-4b'/> for the ovs port. I checked that the ovs port was created by running ovs-vsctl show and found that this time it was created as

Port "tapa65c9a88-4b"
     Interface "tapa65c9a88-4b"

were it should be

Port "tapa65c9a88-4b"
     tag: 2
     Interface "tapa65c9a88-4b"

So I used ovs-vsctl to remove the port and create a new one with the same name and with a correct vlan tag - this didn't work and there was no change.
Rebooting just the nova-compute server didn't fix the problem either (it runs as a nova-compute host, neutron-dhcp host and a neutron-l3 host) The only way to temporarily fix the problem seems to be to reboot all the openstack servers.

edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by koolhead17
close date 2014-06-17 07:53:44.936215

Comments

it is nova-compute that creates the ovs ports for instances. So look in its log for errors.

darragh-oreilly gravatar imagedarragh-oreilly ( 2014-03-14 04:09:49 -0500 )edit

when you create an instance, check that the neutron client correctly shows a newly created port. You may also want to check the libvirt logs for virtio messages.

Tim gravatar imageTim ( 2014-03-18 22:50:09 -0500 )edit

Grep the ovs agent logs for tapa65c9a88-4b. The neutron ovs agent should detect the new ovs port and tag it. I would turn on debug tracing and restart the agent, then check the logs for the reason this is not happening.

darragh-oreilly gravatar imagedarragh-oreilly ( 2014-06-06 04:57:04 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2014-06-08 19:36:55 -0500

zxc123 gravatar image

The problem seemed to be caused by ntp not being setup correctly - the server's clocks were ~1min out from each other which the RPC service (QPID) didn't like. I got all the clocks in sync and the problem hasn't happened since.
For some reason RPC errors only started showing up in the logs after a recent software update.

edit flag offensive delete link more
0

answered 2014-06-05 12:08:46 -0500

rbowen gravatar image

So, following up ... did you get this solved? Did you find anything from the log files, as suggested by @Tim?

edit flag offensive delete link more

Comments

Hey rbowen, that's not an answer :)

larsks gravatar imagelarsks ( 2014-06-05 12:20:00 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-03-13 16:39:50 -0500

Seen: 210 times

Last updated: Jun 08 '14