Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Nova failed to spawn when download image from Glance timeout

Hi all. I recently deploy Bare-metal node instance using Heat Template. However, I encountered a timeout error. When I look into the code I found that the timeout is related when Nova download image from Glance. The nova-schedule.log shows below:

2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last): 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info, 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'], 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks: 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out")) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out

I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?

Thank you very much

Nova failed to spawn when download disk image from Glance timeout

Hi all. I recently deploy Bare-metal node instance using Heat Template. However, I encountered a timeout error. When I look into the code I found that the timeout is related when to Nova download downloading disk image from Glance. The nova-schedule.log shows below:

2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn spawn

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last): last):

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info, network_info=network_info,

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info) tftp_image_info)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'], project_id=instance['project_id'],

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id) project_id)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size) max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size) max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path) dst_path=path)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks: image_chunks:

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE) self._resp.read(CHUNKSIZE)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt) self.fp.read(amt)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left) self._sock.recv(left)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out")) out"))

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch() hub.switch()

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out

I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?

Thank you very much

Nova failed to spawn when download disk image from Glance timeout

Hi all. I recently deploy Bare-metal node instance using Heat Template. However, I encountered a timeout error. When I look into the code I found that the timeout is related to Nova downloading disk image from Glance. The nova-schedule.log shows below:

2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last):

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info,

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'],

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks:

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next() self.next()

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out"))

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch()

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out

I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?

Thank you very much

Nova failed to spawn when download disk image from Glance timeout

Hi all. I recently deploy Bare-metal node instance using Heat Template. However, I encountered Nova failed to spawn due to a timeout error. When I look into the code I found that the timeout is related to Nova downloading disk image from Glance. The nova-schedule.log shows below:

2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last):

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info,

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'],

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks:

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next()

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left)

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out"))

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch()

2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out

I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?

Thank you very much

click to hide/show revision 5
No.5 Revision

Nova failed to spawn when download disk image from Glance timeout

Hi all. I recently deploy Bare-metal node instance using Heat Template. However, Nova failed to spawn due to a timeout error. When I look into the code I found that the timeout is related to Nova downloading disk image from Glance. The nova-schedule.log nova-schedule.log shows below:

2014-02-28 02:49:48.046 2136 ERROR nova.compute.manager [req-09e61b23-436f-4425-8db0-10dd1aea2e39 85bbc1abb4254761a5452654a6934b75 692e595702654930936a65d1a658cff4] [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Instance failed to spawn

spawn 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] Traceback (most recent call last):

last): 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn/ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] network_info=network_info,

network_info=network_info, 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 444, in cache_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] self._cache_tftp_images(context, instance, tftp_image_info)

tftp_image_info) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/pxe.py", line 335, in _cache_tftp_images 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] project_id=instance['project_id'],

project_id=instance['project_id'], 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/baremetal/utils.py", line 33, in cache_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] user_id, project_id)

project_id) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 645, in fetch_image 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

max_size=max_size) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 196, in fetch_to_raw 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] max_size=max_size)

max_size=max_size) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 190, in fetch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] image_service.download(context, image_id, dst_path=path)

dst_path=path) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 360, in download 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] for chunk in image_chunks:

image_chunks: 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 478, in __iter__ 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self.next()

self.next() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 494, in next 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] chunk = self._resp.read(CHUNKSIZE)

self._resp.read(CHUNKSIZE) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/httplib.py", line 561, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] s = self.fp.read(amt)

self.fp.read(amt) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/socket.py", line 380, in read 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] data = self._sock.recv(left)

self._sock.recv(left) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout_exc=socket.timeout("timed out"))

out")) 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 155, in trampoline 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return hub.switch()

hub.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] return self.greenlet.switch() 2014-02-28 02:49:48.046 2136 TRACE nova.compute.manager [instance: 35d00082-4cb4-45a3-a73b-b72ef2a6e2b2] timeout: timed out

out

I don't know where the timeout is set. And does Nova download image from Glance in parallel or in sequential way?

Thank you very much