Ask Your Question
0

instance creating-launching failed:Exhausted all hosts available for retrying build failures for instance

asked 2019-10-20 09:03:04 -0600

rhinolt gravatar image

dashboard has error popup:

Exhausted all hosts available for retrying build failures for instance

and ive check /var/log,assumed the issue was on nova service:/var/log/nova/nova-conductor.log

2019-10-20 20:17:54.725 4474 ERROR nova.scheduler.utils [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Error from last host: controller.lan (node controller): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1984, in _do_build_and_run_instance\n    filter_properties, request_spec)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2354, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: \\u5b9e\\u4f8b96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d\\u7684\\u6784\\u5efa\\u5df2\\u88ab\\u91cd\\u65b0\\u5b89\\u6392\\uff1a\\u7ed1\\u5b9a\\u7aef\\u53e3b4ad70a3-d748-4ba5-9b48-3c469175cd8b\\u5931\\u8d25\\uff0c\\u66f4\\u591a\\u7ec6\\u8282\\u8bf7\\u67e5\\u770b\\u65e5\\u5fd7\\u3002\n']
2019-10-20 20:17:54.728 4474 WARNING nova.scheduler.utils [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] Failed to compute_task_build_instances: 超过最大尝试次数。Exhausted all hosts available for retrying build failures for instance 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d.: MaxRetriesExceeded: \u8d85\u8fc7\u6700\u5927\u5c1d\u8bd5\u6b21\u6570\u3002Exhausted all hosts available for retrying build failures for instance 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d.
2019-10-20 20:17:54.728 4474 WARNING nova.scheduler.utils [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Setting instance to ERROR state.: MaxRetriesExceeded: \u8d85\u8fc7\u6700\u5927\u5c1d\u8bd5\u6b21\u6570\u3002Exhausted all hosts available for retrying build failures for instance 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d.
2019-10-20 20:17:55.153 4474 WARNING oslo_config.cfg [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.).  Its value may be silently ignored in the future.
edit retag flag offensive close merge delete

Comments

There must be more useful info in the compute log.

Do all instances launches fail like that?

Please share some details of the cloud: How it was deployed, how many nodes, version etc.

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-20 10:27:36 -0600 )edit

it is rdo,all in one packstack,on 2 core intel xeon with 16GRAM,centos7. Was fine with first run after installation,built an instance(x-1) with builtin cirros image and all goes ok. BUT all went wrong AFTER reboot,none instance could create(x-2,3...) anymore,the status on instance has error flag.

rhinolt gravatar imagerhinolt ( 2019-10-20 19:30:20 -0600 )edit

also,that instance create before (x-1) could run normally. new volume could create with instance,but instance itself(x-2,3....) went wrong

rhinolt gravatar imagerhinolt ( 2019-10-20 19:35:08 -0600 )edit

What's in the compute log?

You seem to launch your instance with volume as root disk. Same problem with ephemeral storage?

Use openstack compute service list, openstack network agent list, openstack volume service list, systemctl status to check if a component failed to start after reboot.

Bernd Bausch gravatar imageBernd Bausch ( 2019-10-20 21:22:53 -0600 )edit

that nova-compute.log has much more error messages and i dunno how to put them up

rhinolt gravatar imagerhinolt ( 2019-10-21 02:54:51 -0600 )edit

2 answers

Sort by » oldest newest most voted
0

answered 2019-10-21 22:30:24 -0600

rhinolt gravatar image

updated 2019-10-22 03:15:07 -0600

2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] Instance failed network setup after 1 attempt(s): PortBindingFailed: \u7ed1\u5b9a\u7aef\u53e3b4ad70a3-d748-4ba5-9b48-3c469175cd8b\u5931\u8d25\uff0c\u66f4\u591a\u7ec6\u8282\u8bf7\u67e5\u770b\u65e5\u5fd7\u3002 2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager Traceback (most recent call last): 2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1521, in _allocate_network_async 2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager resource_provider_mapping=resource_provider_mapping) 2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1124, in allocate_for_instance

    ......
    (alot File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ etcetc)
    ......

    2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager     raise exception.PortBindingFailed(port_id=port['id'])
    2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager PortBindingFailed: \u7ed1\u5b9a\u7aef\u53e3b4ad70a3-d748-4ba5-9b48-3c469175cd8b\u5931\u8d25\uff0c\u66f4\u591a\u7ec6\u8282\u8bf7\u67e5\u770b\u65e5\u5fd7\u3002
    2019-10-20 20:17:15.808 3276 ERROR nova.compute.manager
    2019-10-20 20:17:46.969 3276 INFO nova.virt.libvirt.driver [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Creating image
    2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Instance failed to spawn: PortBindingFailed: \u7ed1\u5b9a\u7aef\u53e3b4ad70a3-d748-4ba5-9b48-3c469175cd8b\u5931\u8d25\uff0c\u66f4\u591a\u7ec6\u8282\u8bf7\u67e5\u770b\u65e5\u5fd7\u3002
    2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Traceback (most recent call last):


>   ......
>     (again 2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager
> [instance:
> 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] 
> File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line 3199, in spawn like this
> blahblahblah)
>     ......

    2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d]     raise exception.PortBindingFailed(port_id=port['id'])
    2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] PortBindingFailed: \u7ed1\u5b9a\u7aef\u53e3b4ad70a3-d748-4ba5-9b48-3c469175cd8b\u5931\u8d25\uff0c\u66f4\u591a\u7ec6\u8282\u8bf7\u67e5\u770b\u65e5\u5fd7\u3002
    2019-10-20 20:17:46.971 3276 ERROR nova.compute.manager [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d]
    2019-10-20 20:17:46.973 3276 INFO nova.compute.manager [req-dba0151f-9b3b-4c7d-bb19-bd89e230756a e706a94fdf324aecb5794718ed8025b7 b7e49cd82ec645a5a41b7ee841a71aa4 - default default] [instance: 96f2a386-c3b7-4f66-8c28-55bf0c8ffc4d] Terminating instance
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-10-20 08:47:22 -0600

Seen: 62 times

Last updated: Oct 22