Ask Your Question
3

Instance creation fails (Block Device Mapping)

asked 2015-03-10 12:09:34 -0600

lnxslck gravatar image

I'm using Openstack 1.0.1 on Centos 7 64bits, with 3 controllers, 2 compute nodes and storage mounted through nfs.

On Horizon i'm trying to launch a new instance (Boot from image - creates a new volume), and i keep getting the error:

Error: Failed to launch instance "windows_test2": Please try again later [Error: Build of instance e541f467-9a49-4fbf-b8ae-a792c389aa98 aborted: Failure prepping block device.].

  1. The new volume is created and it's OK
  2. An IP is assigned to the instance.
  3. Instance creation stops and shows status ERROR

The status displays Block device mapping for a long time, until it stops and gives the error presented above.

I've tried launching the instance through the command line but the error remains.

On the controller node cinder logs (volume and api) reports no error (logs are with debug=True).

On the compute node nova-compute.log shows:

2015-03-10 16:51:21.938 72172 ERROR nova.compute.manager [-] [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] Instance failed block device setup
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] Traceback (most recent call last):
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1819, in _prep_block_device
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     do_check_attach=do_check_attach) +
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 407, in attach_block_devices
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     map(_log_and_attach, block_device_mapping)
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 405, in _log_and_attach
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     bdm.attach(*attach_args, **attach_kwargs)
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 333, in attach
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     wait_func(context, vol['id'])
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1263, in _await_block_device_map_created
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     attempts=attempts)
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] VolumeNotCreated: Volume 31d43244-7f63-4088-82de-6a521c384404 did not finish being created even after we waited 254 seconds or 61 attempts.
2015-03-10 16:51:21.938 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] 
2015-03-10 16:51:21.939 72172 ERROR nova.compute.manager [-] [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] Failure prepping block device
2015-03-10 16:51:21.939 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98] Traceback (most recent call last):
2015-03-10 16:51:21.939 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2015-03-10 16:51:21.939 72172 TRACE nova.compute.manager [instance: e541f467-9a49-4fbf-b8ae-a792c389aa98]     block_device_mapping)
2015-03-10 16:51 ...
(more)
edit retag flag offensive close merge delete

Comments

Is cinder-volume running on more then one controller?, If so try turning that service off on 2 controllers and see if problem still persists

jdexter gravatar imagejdexter ( 2015-03-12 12:54:14 -0600 )edit

@jdexter Thank you for your help. Yes i had cinder-volume running on all three controllers, i shut it down on two controllers, tried again, and the same error happened. Got more clues?

lnxslck gravatar imagelnxslck ( 2015-03-13 04:56:58 -0600 )edit

What is in your cinder logs on the storage node and what about your cinder API logs on the controller?

ethode gravatar imageethode ( 2015-04-02 18:12:17 -0600 )edit
add a comment

6 answers

Sort by ยป oldest newest most voted
3

answered 2016-02-23 15:27:06 -0600

Radhakrishnan Rk gravatar image

updated 2016-02-24 16:06:56 -0600

Eduardo Gonzalez gravatar image

Hi,

Increase the block device mapping the time out value in nova-compute.conf file. If the size of the volume/image is too large, It will take time to copy. If the timeout value exceeds it will result in failure.

Refer nova.conf reference

block_device_allocate_retries = 60 (default) to 300

block_device_allocate_retries_interval = 3(default) to 10 

block_device_creation_timeout = 10(default) to 300

Best Regards,

Radhakrishnan R

edit flag offensive delete link more

Comments

I'm running Mitaka now and this was the right answer in my case, although I had to make the same changes to the nova.conf on the controller node as well.

eblock gravatar imageeblock ( 2016-05-17 10:40:18 -0600 )edit

Also on Mitaka and these changes made the difference! In the logs: 5 minutes needed for a volume to be created (cinder-volume.log) but after 3 minutes the Block Device is Invalid-error (nova-compute.log).

block_device_creation_timeout is for xenserver only i think

CMIOpenStackAdmin gravatar imageCMIOpenStackAdmin ( 2016-07-11 08:51:14 -0600 )edit

@Radhakrishnan RK@eblock I'm also facing the same issue. I'm beginner to Openstack, whether I should add those 3 lines manually or it is present in nova.conf defaultly?

Harsha gravatar imageHarsha ( 2017-04-18 04:10:16 -0600 )edit
add a comment
1

answered 2016-07-27 04:57:12 -0600

HiFire gravatar image

Hi,

I've had the same issue, but did some steps to get it to work.

  1. Create an image from your distribution (Ubuntu, CentOS, etc.)
  2. Create an instance
  3. Wait for error message
  4. Look into "Volumes"
  5. If the copy process isn't complete wait for it.
  6. Delete the previous instance but not the volume.
  7. Click on the volume and create an instance from it.
  8. Now your instance schould be up and running.
edit flag offensive delete link more
add a comment
0

answered 2016-07-27 14:22:25 -0600

Zachariah gravatar image

is block_device_creation_timeout valid for anything other than Xen?

edit flag offensive delete link more
add a comment
0

answered 2016-04-02 01:52:39 -0600

rackerstacker gravatar image

Even I am facing the same issue. I am running Liberty with Single controller. I am able to create a volume (from cli) and launch instance from an image. But when I try to attach a volume to the instance I get the following error :

[root@controller-01 logs]# nova boot --flavor rhel6.medium --key-name mykey --block-device-mapping vda=f78f46ed-aed6-48ba-bcaf-ea75a717a3be:::0 boot-from-vol-test ERROR (BadRequest): Block Device Mapping is Invalid: failed to get volume f78f46ed-aed6-48ba-bcaf-ea75a717a3be. (HTTP 400) (Request-ID: req-4486ebae-2bc0-48c3-9d8b-29ced20c9261) [root@controller-01 logs]#

edit flag offensive delete link more
add a comment
0

answered 2016-02-23 06:36:05 -0600

Skylab gravatar image

updated 2016-02-29 15:07:12 -0600

Eduardo Gonzalez gravatar image

Hi All

Even I am also facing same issue. But I checked cinder-volume running is running on single node ony. Can you help on this.

Nova Log is below. 

2016-02-23 07:21:34.110 23 INFO nova.osapi_compute.wsgi.server [req-c1ab3b47-d2df-4364-b4fc-34d9cebb6a8e eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/extensions HTTP/1.1" status: 200 len: 22486 time: 2.4953480
/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:442: Warning: Incorrect datetime value: '2016-02-01 00:00:00+00:00' for column 'terminated_at' at row 1
  cursor.execute(statement, parameters)
/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:442: Warning: Incorrect datetime value: '2016-02-23 07:21:34+00:00' for column 'launched_at' at row 1
  cursor.execute(statement, parameters)
2016-02-23 07:21:34.537 23 INFO nova.osapi_compute.wsgi.server [req-6704e6f0-d0c5-4b07-acff-61aed9c24df8 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/os-simple-tenant-usage?start=2016-02-01T00:00:00&end=2016-02-23T23:59:59&detailed=1 HTTP/1.1" status: 200 len: 217 time: 0.4085770
2016-02-23 07:21:34.593 23 INFO nova.osapi_compute.wsgi.server [req-eeacc08b-2aeb-40b6-8f57-195290416a22 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/limits HTTP/1.1" status: 200 len: 711 time: 0.0513749
2016-02-23 07:22:09.529 25 INFO nova.osapi_compute.wsgi.server [req-292f7a33-d99a-4313-8605-6f8d6a199d7f eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/servers/detail?limit=21&project_id=97bca548d2d04e7b85a5c1aeadc0e7cb HTTP/1.1" status: 200 len: 2489 time: 1.5793769
2016-02-23 07:22:10.989 22 INFO nova.osapi_compute.wsgi.server [req-f3ffbef3-ba3a-4ff2-83ec-aa35a0ba24aa eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/flavors/detail HTTP/1.1" status: 200 len: 2297 time: 1.2844250
2016-02-23 07:22:19.252 23 INFO nova.osapi_compute.wsgi.server [req-e530c691-9ba9-448f-9d19-1c3e76859f3f eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/limits?reserved=1 HTTP/1.1" status: 200 len: 711 time: 0.0305040
2016-02-23 07:22:19.279 22 INFO nova.osapi_compute.wsgi.server [req-ea126104-ffd6-4c2a-85f0-4e2e45284929 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/os-hypervisors/detail HTTP/1.1" status: 200 len: 1291 time: 0.0229180
2016-02-23 07:22:20.381 24 INFO nova.osapi_compute.wsgi.server [req-a66ff537-8c5f-4684-bff2-0506a0941f33 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/extensions HTTP/1.1" status: 200 len: 22486 time: 1.0716231
2016-02-23 07:22:20.643 24 INFO nova.osapi_compute.wsgi.server [req-61b2657b-5bf0-462c-b9ad-1c7f8cc210e4 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/os-hypervisors/detail HTTP/1.1" status: 200 len: 1291 time: 0.2494130
2016-02-23 07:22:20.699 24 INFO nova.osapi_compute.wsgi.server [req-dfae1eaa-4e6c-45b9-8222-c5203ebd36c4 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/limits?reserved=1 HTTP/1.1" status: 200 len: 711 time: 0.0319870
2016-02-23 07:22:20.723 22 INFO nova.osapi_compute.wsgi.server [req-8c85f66d-8b58-473a-86c1-7ba346e98694 eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/os-hypervisors/detail HTTP/1.1" status: 200 len: 1291 time: 0.0200250
2016-02-23 07:30:12.380 22 INFO nova.osapi_compute.wsgi.server [req-cf564ea7-6bd1-4c99-88e7-db907fb14a5c eda50140f7a84404af3db6c007e482b3 97bca548d2d04e7b85a5c1aeadc0e7cb - - -] 10.86.156.145 "GET /v2/97bca548d2d04e7b85a5c1aeadc0e7cb/servers/e48ca233-3553-4a0a-82de-fb61f80fda8f HTTP/1.1" status: 200 len: 2486 time: 0.2452590
2016-02-23 07:30 ...
(more)
edit flag offensive delete link more
add a comment
0

answered 2016-10-11 00:19:20 -0600

janice gravatar image

HiFire's answer works for me

edit flag offensive delete link more
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2015-03-10 12:09:34 -0600

Seen: 28,184 times

Last updated: Jul 27 '16

Related questions

Block Device Mapping is Invalid.

creating multiple instances

Block device mapping invalid

Error:Exceeded maximum number of retries,exceeded max scheduling attempts 3 for instance [closed]

When creating an instance from a specific image, I get a failure.

How to connect instance directly to ext-net(external network)?

can't launch an instance - no network

metadata for instances that boot from volumes

Policy doesn't allow compute:create:forced_host to be performed

Cinder: Lost /dev/disks/by-path links