Ask Your Question
0

Error Booting VMs in Grizzly: Instance Exists

asked 2013-08-13 09:52:02 -0500

WiFIGI gravatar image

updated 2013-08-13 13:19:36 -0500

I am trying to boot a couple hundred VMs using Grizzly. I have sufficient resources in my cluster to do this. But when I start booting them, they will randomly (ie. this can happen at the 10th VM or on the 100th) all start failing with this error:

 {u'message': u'InstanceExists', u'code': 500, u'details': u'Instance instance-00000092 already exists. 
   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function     
           return function(self, context, *args, **kwargs)                                                    
       File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1224, in run_instance          
           do_run_instance()                                                                                  
      File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner       
            retval = f(*args, **kwargs)                                                                        
      File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1223, in do_run_instance       
            admin_password, is_first_time, node, instance)                                                     
         File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 879, in _run_instance          
               self._set_instance_error_state(context, instance[\'uuid\'])                                        
         File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__                                       
                self.gen.next()                                                                                   
         File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 790, in _run_instance          
              self._check_instance_exists(context, instance)                                                     
         File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 988, in _check_instance_exists 
             raise exception.InstanceExists(name=instance[\'name\'])  ', u'created': u'2013-08-13T14:38:24Z'}

Once this happens, I can't boot any more VMs regardless of their name (there were no user-readable names conflicting to begin with). I have tried restarting all the services and manually wiping all instance data from the database, but this does not seem to affect the error. The only solution I have found is to completely rebuild my controller node.

Edit: I figured out why this was happening but not how to prevent it. The previous VMs are not being deleted properly, even though nova says they're gone. I have to manually delete them using the commands here: http://ashokcloud.blogspot.com/2012/03/unable-to-run-instance-ends-with-status.html This shouldn't be happening, though, and it only happens sporadically, which makes it hard to diagnose. I am deploying VMs using the Python API and terminating them through the Horizon GUI - is one of these things possibly causing Openstack to mess up?

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
2

answered 2013-08-20 01:43:15 -0500

unmesh-gurjar gravatar image

The root cause of this issue is instance definition exists on hypervisor but the nova database is not aware of it. So, I would suggest either of the following approaches:

  1. do not clear Nova database: do not manually change the Nova database. This would ensure that for instance(s) spawned hence forth, the instance-id begins after the highest known one, avoiding this scenario.
  2. clear Nova database and hypervisor definitions: if you want to wipe out the Nova database yourself, you should also be ensuring that the hypervisor is in sync ( by doing 'virsh undefine' for relevant instances).
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2013-08-13 09:52:02 -0500

Seen: 459 times

Last updated: Aug 20 '13