Yes, I am waiting for this to show up again. We see this in the automated infrastructure. Unfortunately in our setup it's unpredictable, but has shown up fairly often in the past.

Thanks for the update Sumit.

Not sure if you saw my comments above, but it might be good to insert some logging statements into the nova code that talks to quantum, otherwise i'm not sure that we'll have much more information to go on than the info already posted in the question. The quantum logs would be useful as well.

Dan

On Tue, Apr 2, 2013 at 4:06 PM, Sumit Naiksatam < question225158@answers.launchpad.net > wrote:

Question #225158 on quantum changed: https://answers.launchpad.net/quantum/+question/225158 (https://answers.launchpad.net/quantum...)

Sumit Naiksatam posted a new comment: This happened again today, however the automation run got wiped out after the test failure (on account of the dual IPs). We have changed the automation script so that this does not happen the next time. Will investigate once we reproduce this again and have live system. Sorry, I am not providing too much additional information, but I just wanted to report back.

---

You received this question notification because you are an answer contact for quantum.

~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: http://www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here is the nova log as we reproduced this: http://paste.openstack.org/show/35783/

Sorry Dan, I could not add any additional log statements in this run, but I am looking at the setup.

Ok, could you possibly post the portions of the relevant nova-compute + quantum logs, along with a port-show for each of the duplicate ports (or some other way of us seeing the UUIDs of the duplicate ports, and the instance id of the vm)? thanks.

On Thu, Apr 11, 2013 at 12:36 AM, Sumit Naiksatam < question225158@answers.launchpad.net > wrote:

Question #225158 on quantum changed: https://answers.launchpad.net/quantum/+question/225158 (https://answers.launchpad.net/quantum...)

Sumit Naiksatam posted a new comment: Here is the nova log as we reproduced this: http://paste.openstack.org/show/35783/

Sorry Dan, I could not add any additional log statements in this run, but I am looking at the setup.

---

You received this question notification because you are an answer contact for quantum.

~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dan Wendlandt Nicira, Inc: http://www.nicira.com twitter: danwendlandt ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Yes, there is an exception due to time out, and the earlier VM creation failed. But this is the log from the same node, so I am not sure how scheduling on another HV plays into this.

To me, the basic question is why is nova creating two interfaces for this VM (if you see the libvirt conf for this VM). It's not the case that nova creates one interface, but results in two ports being on the Quantum side. Nova is actually creating two interfaces and requesting two ports for those.

Sumit, this is because the order is actually the other way around. Think of it as being that nova contacts quantum to create ports, then when the nova virt layer is actually creating the VM, it looks at how many ports exist for instance X, and creates that many items in the XML. Since both ports are created with the same instance-id as the port's device_id, the underlying virt layer is tricked into thinking there are supposed to be two interfaces.

Okay, but why is nova requesting that two ports be created just in this case when it should be asking for only one port?

Interesting, your nova-compute log shows there was a timeout. Vish was saying that he was thinking perhaps the port was already created (a timeout) vm rescheduled on another HV -- another port created.

I know that we're chiming in on this a bit late, but we recently ran into this issue in our (grizzly+, nvp backed) environment as well.

Thanks for the patch ( https://review.openstack.org/#/c/32691/ ). We will experiment with the new configuration settings and see if this helps us avoid doubly-allocated ports.

I'd like to note that the patch above seems to only be a means of working around the underlying issue -- which AFAICT is coordination of state between nova and neutron. The failure of a discrete api call due to a timeout or some other issue should always be expected and duplicate/retry api calls should be able to be handled in an idempotent manner by the resource manager. Maybe I missed some part of the discussion where this was called out as a larger issue that needs to be addressed and the patch above is just an expedient fix, or perhaps I could be entirely off-base with my understanding of the issue.

Are there any ongoing (or even dusty!) discussions around the way state is managed between openstack services?