Revision history [back]

click to hide/show revision 1
initial version

Windows Instance - Instance failed network setup after 1 attempt(s)”.

Hi,

I have managed to get my Hyper-V instance configured using the CloudBase installer and it shows healthy in the openstack services list. I’m able to deploy VMs on it. The problem is that instance deployment fails with “Instance failed network setup after 1 attempt(s)”.

Checking the Hyper-V logs, I see that a VM is created, at the same time I spawn a new instance thru OpenStack, but it’s shortly deleted. I can also see that the VHD I used as a volume for the OpenStack Instance is copied to the Hyper-V server’s local storage.

The Hyper-V host is Windows Server 2012 R2, I’m using nova network (not neutron). The instance deployment fails exactly at the same place every time. Please find below the nova’s log.

2014-04-01 03:32:02.039 6760 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager Traceback (most recent call last): 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1244, in _allocate_network_async 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager dhcp_options=dhcp_options) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 94, in wrapped 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager return func(self, context, args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 49, in wrapper 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager res = f(self, context, args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 301, in allocate_for_instance 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager nw_info = self.network_rpcapi.allocate_for_instance(context, *args) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\rpcapi.py", line 184, in allocate_for_instance 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager macs=jsonutils.to_primitive(macs)) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\rpcclient.py", line 85, in call 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager return self._invoke(self.proxy.call, ctxt, method, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\rpcclient.py", line 63, in _invoke 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager return cast_or_call(ctxt, msg, *self.kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\openstack\common\rpc\proxy.py", line 130, in call 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager exc.info, real_topic, msg.get('method')) 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>" 2014-04-01 03:32:02.039 6760 TRACE nova.compute.manager 2014-04-01 03:32:02.039 6760 ERROR nova.virt.hyperv.vmops [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>" 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops Traceback (most recent call last): 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 222, in spawn 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops root_vhd_path, eph_vhd_path) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 266, in create_instance 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops for vif in network_info: 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 375, in __iter__ 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return self._sync_wrapper(fn, *args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 366, in _sync_wrapper 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops self.wait() 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\model.py", line 398, in wait 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops self[:] = self._gt.wait() 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\greenthread.py", line 168, in wait 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return self._exit_event.wait() 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\event.py", line 116, in wait 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return hubs.get_hub().switch() 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\hubs\hub.py", line 187, in switch 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return self.greenlet.switch() 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\eventlet\greenthread.py", line 194, in main 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops result = function(args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1244, in _allocate_network_async 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops dhcp_options=dhcp_options) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 94, in wrapped 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return func(self, context, args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 49, in wrapper 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops res = f(self, context, args, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\api.py", line 301, in allocate_for_instance 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops nw_info = self.network_rpcapi.allocate_for_instance(context, *args) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\network\rpcapi.py", line 184, in allocate_for_instance 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops macs=jsonutils.to_primitive(macs)) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\rpcclient.py", line 85, in call 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return self._invoke(self.proxy.call, ctxt, method, *kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\rpcclient.py", line 63, in _invoke 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops return cast_or_call(ctxt, msg, *self.kwargs) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\openstack\common\rpc\proxy.py", line 130, in call 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops exc.info, real_topic, msg.get('method')) 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>" 2014-04-01 03:32:02.039 6760 TRACE nova.virt.hyperv.vmops 2014-04-01 03:32:02.055 6760 INFO nova.virt.hyperv.vmops [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] Got request to destroy instance: instance-00000117 2014-04-01 03:32:02.664 6760 WARNING nova.openstack.common.rpc.amqp [-] No calling threads waiting for msg_id : 38d6deb785ca4dd6ac7782cbb5dfdc78, message : {u'_unique_id': u'd84ccf34654b44378a185832e641e86c', u'failure': u'{"args": ["Timeout while waiting on RPC response - topic: \"network.compute2\", RPC method: \"_rpc_allocate_fixed_ip\" info: \"<unknown>\""], "module": "nova.openstack.common.rpc.common", "kwargs": {"info": "<unknown>", "topic": "network.compute2", "method": "_rpc_allocate_fixed_ip"}, "message": "Timeout while waiting on RPC response - topic: \"network.compute2\", RPC method: \"_rpc_allocate_fixed_ip\" info: \"<unknown>\"", "tb": ["Traceback (most recent call last):\n", " File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py\", line 461, in _process_data\n *args)\n", " File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py\", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, *kwargs)\n", " File \"/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py\", line 118, in allocate_for_instance\n *kwargs)\n", " File \"/usr/lib/python2.7/dist-packages/nova/network/manager.py\", line 510, in allocate_for_instance\n requested_networks=requested_networks)\n", " File \"/usr/lib/python2.7/dist-packages/nova/network/manager.py\", line 217, in _allocate_fixed_ips\n gt.wait()\n", " File \"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py\", line 168, in wait\n return self._exit_event.wait()\n", " File \"/usr/lib/python2.7/dist-packages/eventlet/event.py\", line 116, in wait\n return hubs.get_hub().switch()\n", " File \"/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py\", line 187, in switch\n return self.greenlet.switch()\n", " File \"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py\", line 194, in main\n result = function(args, *kwargs)\n", " File \"/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py\", line 295, in _rpc_allocate_fixed_ip\n address=address, vpn=vpn)\n", " File \"/usr/lib/python2.7/dist-packages/nova/rpcclient.py\", line 85, in call\n return self._invoke(self.proxy.call, ctxt, method, *kwargs)\n", " File \"/usr/lib/python2.7/dist-packages/nova/rpcclient.py\", line 63, in _invoke\n return cast_or_call(ctxt, msg, *self.kwargs)\n", " File \"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py\", line 130, in call\n exc.info, real_topic, msg.get(\'method\'))\n", "Timeout: Timeout while waiting on RPC response - topic: \"network.compute2\", RPC method: \"_rpc_allocate_fixed_ip\" info: \"<unknown>\"\n"], "class": "Timeout"}', u'result': None} 2014-04-01 03:32:02.664 6760 WARNING nova.openstack.common.rpc.amqp [-] _call_waiters: {} 2014-04-01 03:32:02.806 6760 ERROR nova.compute.manager [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] [instance: 32f8263e-ebff-4db1-b730-e31c80111704] Instance failed to spawn 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] Traceback (most recent call last): 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1423, in _spawn 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] block_device_info) 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\driver.py", line 55, in spawn 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] admin_password, network_info, block_device_info) 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 88, in wrapper 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] return function(self, args, *kwds) 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 232, in spawn 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] raise vmutils.HyperVException(_('Spawn instance failed')) 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] HyperVException: Spawn instance failed 2014-04-01 03:32:02.806 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] 2014-04-01 03:32:02.884 6760 AUDIT nova.compute.manager [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] [instance: 32f8263e-ebff-4db1-b730-e31c80111704] Terminating instance 2014-04-01 03:32:03.072 6760 INFO nova.virt.hyperv.vmops [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] Got request to destroy instance: instance-00000117 2014-04-01 03:32:03.181 6760 ERROR nova.compute.manager [req-b50eb6a0-74e7-4b19-8e86-5a816ec606f1 3cdb0cfec21546c68fbf4ddf68acf773 0ea17d222eaa44948b0b8f8ed9c5cece] [instance: 32f8263e-ebff-4db1-b730-e31c80111704] Error: Spawn instance failed 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] Traceback (most recent call last): 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1043, in _build_instance 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] set_access_ip=set_access_ip) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1426, in _spawn 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\compute\manager.py", line 1423, in _spawn 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] block_device_info) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\driver.py", line 55, in spawn 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] admin_password, network_info, block_device_info) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 88, in wrapper 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] return function(self, args, *kwds) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\nova\virt\hyperv\vmops.py", line 232, in spawn 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] raise vmutils.HyperVException(_('Spawn instance failed')) 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704] HyperVException: Spawn instance failed 2014-04-01 03:32:03.181 6760 TRACE nova.compute.manager [instance: 32f8263e-ebff-4db1-b730-e31c80111704]