Extra nova volume attachment after cinder reset-state detached

asked 2019-02-28 01:48:19 -0500

Yi Song gravatar image

updated 2019-03-01 07:20:03 -0500

Hi,

cinder reset-state --state available --attach-status detached [Share volume id]

It looks the above command does NOT remove the volume attachment from VM?

In my case, the VM located compute node is down, then I need to detach the cinder volume from the VM, and attach it to another VM.

But "nova volume-detach“ command not work, as the compute node is down. The cinder volume is hung in detaching state.

Then, I run the following commands to reset cinder state, and attach to another VM.

cinder reset-state --state available --attach-status detached [Share volume id]
nova volume-attach [new VM id] [Share volume id]

But "nova volume-attachments" still show the cinder volume attached to original down VM. I can see the extra record in nova.block_device_mapping table.

Added more details: The storage backend is ceph.

/etc/keystone/keystone.conf on OpenStack controller:
[token]
expiration = 3600

If bring up the compute node in token expire timer, the previous "nova volume-detach" command will get processed by computed node, and remove the volume attachment to old VM. Everything is fine.

But if bring up the compute node after token expire timer, the previous "nova volume-detach" command can not be processed, so the volume attachment to old VM be left there, and become dirty record.

Question: Is there any openstack command to remove the extra volume attachment? Not SQL commands to delete it from MySQL database.

Thanks, Yi Song

edit retag flag offensive close merge delete

Comments

If the compute node is down then the services (nova and cinder) can't execute commands, so your API calls can't take effect. What storage backend are you using? Can't you bring the compute node back up?

eblock gravatar imageeblock ( 2019-02-28 06:43:46 -0500 )edit

I'm not sure why the token expiration should take effect here. 1 hour to detach a volume should be quite enough, shouldn't it? As I already said, how could a command over the API work if the compute node is down? The only way I can imagine is the manipulation of the DB. So the answer is probably no.

eblock gravatar imageeblock ( 2019-03-04 02:51:25 -0500 )edit

And if the token expiration really is an issue, can't you just increase it? It's still not quite clear to me what you actually are trying to achieve and why the workarounds don't work for you.

eblock gravatar imageeblock ( 2019-03-04 02:52:29 -0500 )edit