Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Evacuate Failure. (with SharedStorage, over AZ)

Hello, I would like to ask about "Evacuate Failure".

Although I tried to do evacuate a instance from failed host to specified one, it was failed. The instance data is placed on shared storage(Ceph).

Environment

  • Openstack: IceHouse
  • Ceph: Firefly
  • OS: CentOS 6.5

Procedure

Initial State:

# nova hypervisor-list
+----+-----------------------+
| ID | Hypervisor hostname   |
+----+-----------------------+
| 9  | node-28.mng.kvh.ne.jp |
| 12 | node-29.mng.kvh.ne.jp |
+----+-----------------------+

# nova list --field name,status,,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname | egrep "ID|admin_vm02"
| ID                                   | Name       | Status |  | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
| 496e2e97-e167-42e9-a0ce-394eb4fd72db | admin_vm02 | ACTIVE |  | active               | nova2                        | node-29.mng.kvh.ne.jp                |

# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova                     | available                              |
| |- node-28.mng.kvh.ne.jp |                                        |
| | |- nova-compute        | enabled :-) 2015-02-18T04:39:15.000000 |
| nova2                    | available                              |
| |- node-29.mng.kvh.ne.jp |                                        |
| | |- nova-compute        | enabled :-) 2015-02-18T04:39:15.000000 |
+--------------------------+----------------------------------------+

Stop nova-compute(node-29):

# service openstack-nova-compute stop

# nova service-list | egrep "Host|nova-compute"
| Binary           | Host                  | Zone     | Status  | State | Updated_at                 | Disabled Reason |
| nova-compute     | node-28.mng.kvh.ne.jp | nova     | enabled | up    | 2015-02-18T05:37:55.000000 | -               |
| nova-compute     | node-29.mng.kvh.ne.jp | nova2    | enabled | down  | 2015-02-18T05:36:15.000000 | -               |

Evacuate (from node-29 to node-28):

# nova evacuate --on-shared-storage 496e2e97-e167-42e9-a0ce-394eb4fd72db node-28.mng.kvh.ne.jp

Compute Log (node-28):

<182>Feb 18 05:42:48 node-28 nova-compute 2015-02-18 14:42:48.014 7240 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-28.mng.kvh.ne.jp:node-28.mng.kvh.ne.jp
<180>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.251 7240 AUDIT nova.compute.manager [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 None] [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Rebuilding instance
<179>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.253 7240 ERROR nova.compute.manager [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 None] [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Setting instance vm_state to ERROR
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Traceback (most recent call last):
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5564, in _error_out_instance_on_exception
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]     yield
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2444, in rebuild_instance
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]     _("Invalid state of instance files on shared"
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] InvalidSharedStorage: Invalid state of instance files on shared storage
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]
<179>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.538 7240 ERROR oslo.messaging.rpc.dispatcher [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 ] Exception during message handling: Invalid state of instance files on shared storage
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     payload)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 274, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     pass
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 260, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 327, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     function(self, context, *args, **kwargs)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 303, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 290, in decorated_function
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2444, in rebuild_instance
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher     _("Invalid state of instance files on shared"
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher InvalidSharedStorage: Invalid state of instance files on shared storage
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher

Nova Status:

# nova list --field name,status,,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------+--------+--+----------------------+------------------------------+--------------------------------------+
| ID                                   | Name       | Status |  | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------+--------+--+----------------------+------------------------------+--------------------------------------+
| 496e2e97-e167-42e9-a0ce-394eb4fd72db | admin_vm02 | ERROR  |  | error                | nova2                        | node-29.mng.kvh.ne.jp                |
+--------------------------------------+------------+--------+--+----------------------+------------------------------+--------------------------------------+


# nova show 496e2e97-e167-42e9-a0ce-394eb4fd72db | egrep "Property| fault "
| Property                             | Value                                                                                                                                                                                          |
| fault                                | {"message": "Invalid state of instance files on shared storage", "code": 500, "details": "  File \"/usr/lib/python2.6/site-packages/nova/compute/manager.py\", line 290, in decorated_function |

Thank you.