Evacuate Failure. (with SharedStorage, over AZ)
Hello, I would like to ask about "Evacuate Failure".
Although I tried to do evacuate a instance from failed host to specified one, it was failed. The instance data is placed on shared storage(Ceph).
Environment
- Openstack: IceHouse
- Ceph: Firefly
- OS: CentOS 6.5
Procedure
Initial State:
# nova hypervisor-list
+----+-----------------------+
| ID | Hypervisor hostname |
+----+-----------------------+
| 9 | node-28.mng.kvh.ne.jp |
| 12 | node-29.mng.kvh.ne.jp |
+----+-----------------------+
# nova list --field name,status,,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname | egrep "ID|admin_vm02"
| ID | Name | Status | | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
| 496e2e97-e167-42e9-a0ce-394eb4fd72db | admin_vm02 | ACTIVE | | active | nova2 | node-29.mng.kvh.ne.jp |
# nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- node-28.mng.kvh.ne.jp | |
| | |- nova-compute | enabled :-) 2015-02-18T04:39:15.000000 |
| nova2 | available |
| |- node-29.mng.kvh.ne.jp | |
| | |- nova-compute | enabled :-) 2015-02-18T04:39:15.000000 |
+--------------------------+----------------------------------------+
Stop nova-compute(node-29):
# service openstack-nova-compute stop
# nova service-list | egrep "Host|nova-compute"
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| nova-compute | node-28.mng.kvh.ne.jp | nova | enabled | up | 2015-02-18T05:37:55.000000 | - |
| nova-compute | node-29.mng.kvh.ne.jp | nova2 | enabled | down | 2015-02-18T05:36:15.000000 | - |
Evacuate (from node-29 to node-28):
# nova evacuate --on-shared-storage 496e2e97-e167-42e9-a0ce-394eb4fd72db node-28.mng.kvh.ne.jp
Compute Log (node-28):
<182>Feb 18 05:42:48 node-28 nova-compute 2015-02-18 14:42:48.014 7240 INFO nova.compute.resource_tracker [-] Compute_service record updated for node-28.mng.kvh.ne.jp:node-28.mng.kvh.ne.jp
<180>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.251 7240 AUDIT nova.compute.manager [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 None] [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Rebuilding instance
<179>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.253 7240 ERROR nova.compute.manager [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 None] [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Setting instance vm_state to ERROR
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] Traceback (most recent call last):
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5564, in _error_out_instance_on_exception
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] yield
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2444, in rebuild_instance
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] _("Invalid state of instance files on shared"
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db] InvalidSharedStorage: Invalid state of instance files on shared storage
2015-02-18 14:43:02.253 7240 TRACE nova.compute.manager [instance: 496e2e97-e167-42e9-a0ce-394eb4fd72db]
<179>Feb 18 05:43:02 node-28 nova-compute 2015-02-18 14:43:02.538 7240 ERROR oslo.messaging.rpc.dispatcher [req-85f022af-cf7a-46c8-a95d-6e885055fcf7 ] Exception during message handling: Invalid state of instance files on shared storage
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-02-18 14:43:02.538 7240 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-02-18 ...