Ask Your Question
0

Extending an in-use cinder volume?

asked 2019-02-11 04:30:38 -0500

pv2b gravatar image

updated 2019-02-11 04:40:09 -0500

Hello!

I have a recently deployed CentOS 7 VM with OpenStack rocky deployed using the PackStack all-in-one deployment, in order to evaluate and learn about OpenStack.

In the PackStack setup, the LVM Volume Driver is used by Cinder. The Libvirt driver is used by Nova. (The volume_type is shown as iscsi, I'm not sure why, I think this might be misleading.)

Right now I'm investigating the use case of being able to extend a volume that is currently attached to a running instance, in order to provision extra storage space on the fly. As far as I can tell, work was done in 2017 to add this functionality to Pike, as per the following specs:

  • https://specs.openstack.org/openstack/cinder-specs/specs/pike/extend-attached-volume.html (Cinder Specs: Extend a volume while volume is attached to an instance)
  • https://specs.openstack.org/openstack/nova-specs/specs/pike/implemented/nova-support-attached-volume-extend.html (Nova Specs: Allow an attached volume to be extended)

What I'm trying to do is to extend an online volume - i.e. grow the available disk space available to an instance without rebooting it, and without adding an additional volume.

Consulting the https://docs.openstack.org/cinder/rocky/reference/support-matrix.html#operation_online_extend_support (Cinder Driver Support Matrix (Rocky)), I have also been able to confirm that this is supported by the LVM driver.

According to the Nova spec:

The initial implementation aims to support virt drivers using os-brick such as libvirt and hyper-v.

So, as far as I can tell, the configuration I'm running (PackStack all-in-one) should support this functionality.

However, I have not been able to find this functionality exposed anywhere in the OpenStack CLI or Horizon. So, I tried an API call as per the https://developer.openstack.org/api-ref/block-storage/v3/index.html?expanded=extend-a-volume-size-detail#volume-actions-volumes-action (API Documentation (Block Storage volume actions)).

The API call below aims to try to extend the volume that's currently attached to the VM from 1 GB to 2 GB.

$ curl -s -H "Content-Type: application/json" -H "X-Auth-Token: <redacted>" http://<redacted>:8776/v3/a17f01bc783445f5b352209283f4cc02/volumes/006e4fcc-aa4d-414b-981c-5b2360d30202/action -d '{"os-extend":{"new_size":2}}'
{"badRequest": {"message": "Invalid volume: Volume 006e4fcc-aa4d-414b-981c-5b2360d30202 status must be '{'status': 'available'}' to extend, currently in-use.", "code": 400}}

As we can see, an error is thrown by Cinder, indicating that an in-use volume cannot be extended.

What am I missing? Is there something about OpenStack as deployed by Packstack that is incompatible with this functionality - i.e. being able to resize volumes on the fly? Or is the functionality simply not there, despite the documentation to the contrary?

For reference, here's a view of the cinder volume and the nova instance to which it is connected:

[root@localhost ~(keystone_demo)]# cinder show 006e4fcc-aa4d-414b-981c-5b2360d30202
+------------------------------+-------------------------------------------------+
| Property                     | Value                                           |
+------------------------------+-------------------------------------------------+
| attached_servers             | ['1dcd32a3-41d0-443d-abd4-9e054cb1844d']        |
| attachment_ids               | ['e67c034c-b800-42e8-9478-71e712f60ed2']        |
| availability_zone            | nova                                            |
| bootable                     | true                                            |
| consistencygroup_id          | None                                            |
| created_at                   | 2019-02-10T18:16:37.000000                      |
| description                  |                                                 |
| encrypted                    | False                                           |
| id                           | 006e4fcc-aa4d-414b-981c-5b2360d30202            |
| metadata                     | attached_mode : rw                              |
|                              | readonly : False                                |
| multiattach                  | False                                           |
| name                         |                                                 |
| os-vol-tenant-attr:tenant_id | a17f01bc783445f5b352209283f4cc02                |
| readonly                     | False                                           |
| replication_status           | None                                            |
| size                         | 1                                               |
| snapshot_id                  | None                                            |
| source_volid                 | None                                            |
| status                       | in-use                                          |
| updated_at                   | 2019-02-10T18 ...
(more)
edit retag flag offensive close merge delete

Comments

Have you verified that your openstack version installed by packstack is actually Pike or newer? What does the output of nova-manage --version show? Pike version is 16.X.

eblock gravatar imageeblock ( 2019-02-11 07:09:05 -0500 )edit

@eblocknova-manage --version shows 18.1.0. I installed Packstack using the centos-release-openstack-rocky yum package. So, as I wrote in the question, I expect I'm running rocky.

pv2b gravatar imagepv2b ( 2019-02-11 08:14:24 -0500 )edit

Progress! I was missing a version header with the API call. Adding -H "OpenStack-API-Version: volume 3.42 to the curl command gave me a positive response. cinder now shows the volume as 2GB. I can't get Cirros to see the extra space but that might be a cirros thing. I'll try another OS.

pv2b gravatar imagepv2b ( 2019-02-11 13:07:43 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-02-11 14:44:18 -0500

pv2b gravatar image

After some investigation this is what I have learned:

  1. I had the curl call almost right. I realized that I had omitted the OpenStack-API-Version header. The following curl call would work:

    curl -s -H "Content-Type: application/json" -H "X-Auth-Token: <redacted>" -H "OpenStack-API-Version: volume 3.42" http://<redacted>:8776/v3/a17f01bc783445f5b352209283f4cc02/volumes/006e4fcc-aa4d-414b-981c-5b2360d30202/action -d '{"os-extend":{"new_size":2}}'
    
  2. There is in fact an openstack cli tool. It is called like this:

    OS_VOLUME_API_VERSION=3.42 cinder extend 006e4fcc-aa4d-414b-981c-5b2360d30202 2
    

    Again, specifying the API version is neccessary here in order to enable extending of in-use volumes. I still haven't found anything to expose this functionality in Horizon.

  3. This almost works on my packstack setup, but I get the following in my /var/log/cinder/volume.log:

    2019-02-11 19:38:10.807 10278 ERROR cinder.compute.nova [req-7c27e5dc-d7eb-4290-a48f-290fe1964e08 2b2c9c98ce6948e28ba007722a2f9588 a17f01bc783445f5b352209283f4cc02 - default default] Failed to notify nova on events: [{'tag': u'006e4fcc-aa4d-414b-981c-5b2360d30202', 'name': 'volume-extended', 'server_uuid': u'1dcd32a3-41d0-443d-abd4-9e054cb1844d'}].: Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-dec890fe-9c2a-45a3-885c-3f0d3acd369e)
    

    Searching for this, I found a Red Hat bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1640443 (https://bugzilla.redhat.com/show_bug....)

    It seems on my system, I'm missing an admin connection between nova and cinder. This has been fixed in TripleO, but likely not in packstack deployments. However, fixing that's a question for a different question. :-)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2019-02-11 04:30:38 -0500

Seen: 699 times

Last updated: Feb 11