Ask Your Question
0

Nova instance throws an error on launch - "failed to perform requested operation on instance….the server has either erred or is incapable of performing the requested operation (HTTP 500)"

asked 2019-01-31 22:07:13 -0500

Yogesh gravatar image

updated 2019-01-31 22:45:51 -0500

Nova instance throws an error on launch - "failed to perform requested operation on instance….the server has either erred or is incapable of performing the requested operation (HTTP 500)".

Surprisingly it works well when attaching volume separately after instance launch. You need set "Create New Volume” to “No” during creation of instance.

We restarted cinder service, but it did not solve the issue.

From the API logs we figured out that there is HTTP 500 error during API interactions in service endpoints (Nova & Cinder). Logs pasted below.

Can someone help to resolve this issue ?

Thanks in advance.

==> api.log <== 2019-01-30 04:16:28.785 275098 ERROR cinder.api.middleware.fault [req-634abf81-df79-42b5-b8f4-8f19488c0bba a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - default default] Caught error: <class 'oslo_messaging.exceptions.messagingtimeout'=""> Timed out waiting for a reply to message ID bf2f80590a754b59a720405cd0bc1ffb 2019-01-30 04:16:28.785 275098 ERROR cinder.api.middleware.fault Traceback (most recent call last): 2019-01-30 04:16:28.785 275098 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__ 2019-01-30 04:16:28.785 275098 ERROR cinder.api.middleware.fault return req.get_response(self.application) 2019-01-30 04:16:28.785 275098 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2019-01-30 04:16:28.793 275098 INFO cinder.api.middleware.fault [req-634abf81-df79-42b5-b8f4-8f19488c0bba a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - default default] http://10.110.77.2:8776/v2/2db5c111414e4d2bbc14645e6f0931db/volumes/301f71f0-8fb5-4429-a67c-473d42ff9def/action (http://10.110.77.2:8776/v2/2db5c11141...) returned with HTTP 500 2019-01-30 04:16:28.794 275098 INFO eventlet.wsgi.server [req-634abf81-df79-42b5-b8f4-8f19488c0bba a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - default default] 10.110.77.4 "POST /v2/2db5c111414e4d2bbc14645e6f0931db/volumes/301f71f0-8fb5-4429-a67c-473d42ff9def/action HTTP/1.1" status: 500 len: 425 time: 60.0791931 2019-01-30 04:16:28.813 275098 INFO cinder.api.openstack.wsgi [req-53d149ac-6e60-4ddd-9ace-216d12122790 a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - default default] POST http://10.110.77.2:8776/v2/2db5c111414e4d2bbc14645e6f0931db/volumes/301f71f0-8fb5-4429-a67c-473d42ff9def/action (http://10.110.77.2:8776/v2/2db5c11141...) 2019-01-30 04:16:28.852 275098 INFO cinder.volume.api [req-53d149ac-6e60-4ddd-9ace-216d12122790 a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - default default] Volume info retrieved successfully.</class>

Nova Logs : 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [req-a4b94c35-2532-4e82-864c-ff33b972a3b2 a1c4c0232896400ba6fddf1cbcb54dd8 2db5c111414e4d2bbc14645e6f0931db - - -] [instance: aba62cf8-0880-4bf7-8201-3365861c8079] Instance failed block device setup 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] Traceback (most recent call last): 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1588, in _prep_block_device 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] wait_func=self._await_block_device_map_created) 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 512, in attach_block_devices 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] _log_and_attach(device) 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance: aba62cf8-0880-4bf7-8201-3365861c8079] File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 509, in _log_and_attach 2019-01-30 03:58:04.808 5642 ERROR nova.compute.manager [instance ... (more)

edit retag flag offensive close merge delete

Comments

I remember timeout issues between nova & cinder in my first days with openstack. If attaching a volume to a running instance works it sounds like you should check that. Not sure about the options, but something like block_device_creation_timeout and block_device_allocate_retries.

eblock gravatar imageeblock ( 2019-02-01 05:15:40 -0500 )edit

hi, I got the issue resolved today. I have observed there were few projects in Openstack where volume deletion stuck in error state with "Error deleting". I changed the volume state explicitly from cinder db using the "cinder reset-state --state available volume-id".

Yogesh gravatar imageYogesh ( 2019-02-10 22:51:28 -0500 )edit

this allowed me to delete the volume successfully. I restarted the cinder service then after and everything stated working as ususal

Yogesh gravatar imageYogesh ( 2019-02-10 22:52:16 -0500 )edit

1 answer

Sort by » oldest newest most voted
0

answered 2019-02-10 23:07:06 -0500

Yogesh gravatar image

hi, I got the issue resolved today. I have observed there were few projects in Openstack where volume deletion stuck in error state with "Error deleting". I changed the volume state explicitly from cinder db using the "cinder reset-state --state available volume-id".

This allowed me to delete the volume successfully. I restarted the cinder service then after and everything stated working as usual

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-01-31 22:07:13 -0500

Seen: 101 times

Last updated: Jan 31