Ask Your Question
0

Should data in the root drive survive an evacuation

asked 2014-09-08 13:00:16 -0600

AaronTSmith gravatar image

Scenario: Two hypervisors/nodes are setup for shared storage. An instance is running on one of the nodes. Some data is written to the root drive. The running hypervisor fails. and an evacuation is performed. Should the data in the root drive be preserved. The root volume is not a cinder volue

edit retag flag offensive close merge delete

Comments

I have seen several references in the OpenStack literature about evacuation preserving user data if the instances directory is on shared storage.

AaronTSmith gravatar imageAaronTSmith ( 2014-09-09 21:11:13 -0600 )edit

2 answers

Sort by ยป oldest newest most voted
1

answered 2014-10-28 19:19:46 -0600

WZ gravatar image

I traced the nova python code (Grizzly) and found a bug in nova/compute/manager.py. See following code snippets:

def rebuild_instance(self, context, instance, orig_image_ref, image_ref, injected_files, new_pass, orig_sys_metadata, bdms, recreate, on_shared_storage, preserve_ephemeral=False):

        .......

        kwargs = dict(
            context=context,
            instance=instance,
            image_meta=image_meta,
            injected_files=files,
            admin_password=new_pass,
            bdms=bdms,
            detach_block_devices=detach_block_devices,
            attach_block_devices=self._prep_block_device,
            block_device_info=block_device_info,
            network_info=network_info,
            preserve_ephemeral=preserve_ephemeral,
            recreate=recreate)       <---- this parameter is missed in the code
        try:
            self.driver.rebuild(**kwargs)
        except NotImplementedError:
            # NOTE(rpodolyaka): driver doesn't provide specialized version
            # of rebuild, fall back to the default implementation
            self._rebuild_default_impl(**kwargs)     
      .......

def _rebuild_default_impl(self, context, instance, image_meta, injected_files, admin_password, bdms, detach_block_devices, attach_block_devices, network_info=None, recreate=False, block_device_info=None, preserve_ephemeral=False): if preserve_ephemeral: # The default code path does not support preserving ephemeral # partitions. raise exception.PreserveEphemeralNotSupported()

    detach_block_devices(context, bdms)

    if not recreate:     <--- Because "recreate=recreate" is missed in rebuild_instance when building the argument
                                <--- list, the default recreate which is False is taken. self.driver.destroy will delete the 
                                <--- /var/lib/nova/instances/uuid directory and then re-create it.
        self.driver.destroy(context, instance, network_info,
                            block_device_info=block_device_info)

Adding "recreate=recreate" may not be the proper fix but at least it explains why the root drive is deleted.

-WZ

edit flag offensive delete link more
1

answered 2014-09-08 13:13:22 -0600

SamYaple gravatar image

Host Evactuation is database action only. Nova simply reschdules the instance and reattaches storage if it is shared storage. If it is ephemeral, it is lost.

That is my understanding at least. I would highly recommend a test if you are unsure.

edit flag offensive delete link more

Comments

Thanks for the reply.

What I am seeing is that the directory on the shared drive: /var/lib/nova/instances/UUID is deleted and recreated. Why would OpenStack require a shared drive if it were only going to delete all of the information in the shared drive?

AaronTSmith gravatar imageAaronTSmith ( 2014-09-08 17:06:26 -0600 )edit

I see the following errors:

ERROR nova.virt.libvirt.driver instance: 2bc62cee-2d3b-49ae-98c9-5b0c352fc35d] During wait destroy, instance disappeared.

AaronTSmith gravatar imageAaronTSmith ( 2014-09-08 17:07:45 -0600 )edit

unplug /opt/stack/nova/nova/virt/libvirt/vif.py:774 Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ovs-vsctl --timeout=120 del-port br-int qvo7793c72d-9c execute /opt/stack/nova/nova/openstack/common/processutils.py:1 Result was 1 ERROR nova.network.linux_net [ESC[01;36mre

AaronTSmith gravatar imageAaronTSmith ( 2014-09-08 17:07:57 -0600 )edit

I did the same test (evacuate an instance with --on-shared-storage) and saw the same behavior. I was also wondering the same thing why shared-storage is required if directory /var/lib/nova/instances/uuid is deleted and recreated for the evacuated instance anyway. -WZ

WZ gravatar imageWZ ( 2014-10-23 12:27:04 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-09-08 13:00:16 -0600

Seen: 160 times

Last updated: Oct 28 '14