Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Failed to scheduler_run_instance: No valid host was found -- Unathorized: Authentication required

Late last night our AD cert expired and was subsequently renewed (used for authentication). Authentication to our openstack was broken and eventually fixed (ie Users logging into Horizon). During this process we broke something somewhere else. When creating a new instance in any tenant we get the following errors/messages:

HORIZON message: Error: Failed to launch instance "<servername>": Please try again later [Error: No valid host was found. Exceeded max scheuling attmepts 3 for instance <splat>].

nova-scheduler.log 2016-02-01 13:35:01.099 1504 INFO nova.scheduler.filter_scheduler [req-624f79c7-5f42-42df-9801-4824ba8befc7 c3cb803eb8164cc1b72233b248e16847 c91cd07bd5a24c0e8ea69235159a96af] Attempting to build 1 instance(s) uuids: [u'3 bd1ea8e-684c-4977-9831-f6dcb0ad0627'] 2016-02-01 13:35:01.100 1504 ERROR nova.scheduler.filter_scheduler [req-624f79c7-5f42-42df-9801-4824ba8befc7 c3cb803eb8164cc1b72233b248e16847 c91cd07bd5a24c0e8ea69235159a96af] [instance: 3bd1ea8e-684c-4977-9831-f6dcb0ad0 627] Error from last host: vapor15 (node vapor15.ac): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1333, in _build_instance\n set_access_ip=set_ac cess_ip)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 401, in decorated_function\n return function(self, context, *args, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/c ompute/manager.py", line 1745, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1742, in _spawn\n block_device_info)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt /libvirt/driver.py", line 2292, in spawn\n write_to_disk=True)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3473, in to_xml\n network_info_str = str(network_info)\n', u' File " /usr/lib/python2.7/dist-packages/nova/network/model.py", line 424, in __str__\n return self._sync_wrapper(fn, *args, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 407, in _sync_ wrapper\n self.wait()\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 439, in wait\n self[:] = self._gt.wait()\n', u' File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", li ne 168, in wait\n return self._exit_event.wait()\n', u' File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 120, in wait\n current.throw(self._exc)\n', u' File "/usr/lib/python2.7/dist-packages/event let/greenthread.py", line 194, in main\n result = function(args, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1532, in _allocate_network_async\n dhcp_options=dhcp_options )\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 361, in allocate_for_instance\n LOG.exception(msg, port_id)\n', u' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/ex cutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 336, in allocate_for_instance\n security_group_ids , available_macs, dhcp_opts)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 195, in _create_port\n network_id, instance=instance)\n', u' File "/usr/lib/python2.7/dist-packages/nov a/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 184, in _create_port\n port_i d = port_client.create_port(port_req_body)[\'port\'][\'id\']\n', u' File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 111, in with_params\n ret = self.function(instance, *args, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 316, in create_port\n return self.post(self.ports_path, body=body)\n', u' File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/clien t.py", line 1241, in post\n headers=headers, params=params)\n', u' File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1156, in do_request\n resp, replybody = self.httpclient.do_request(acti on, method, body=body)\n', u' File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 199, in do_request\n self.endpoint_url + url, method, kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/neutr onclient/client.py", line 163, in _cs_request\n raise exceptions.Unauthorized(message=body)\n', u'Unauthorized: Authentication required\n'] 2016-02-01 13:35:01.101 1504 WARNING nova.scheduler.utils [req-624f79c7-5f42-42df-9801-4824ba8befc7 c3cb803eb8164cc1b72233b248e16847 c91cd07bd5a24c0e8ea69235159a96af] Failed to scheduler_run_instance: No valid host was f ound. Exceeded max scheduling attempts 3 for instance 3bd1ea8e-684c-4977-9831-f6dcb0ad0627 2016-02-01 13:35:01.102 1504 WARNING nova.scheduler.utils [req-624f79c7-5f42-42df-9801-4824ba8befc7 c3cb803eb8164cc1b72233b248e16847 c91cd07bd5a24c0e8ea69235159a96af] [instance: 3bd1ea8e-684c-4977-9831-f6dcb0ad0627] Sett ing instance to ERROR state.

nova-api.log only seemed to have some generic stuff in it

The nova-scheduler.log gives the key (I think), that authentication is somehow broken. I am new to our environment (fairly new hire) and am therefore unfamiliar with openstack as I would like to be. Could someone point me in the right direction so that I might resolve this? I can personally authenticate into openstack (via horizon) and get into my tenant, but it appears as if authentication is going on during the create-an-instance process. This appears to be the piece (one of many) I am unfamiliar with.

Side-note: Based on my searches the error in the title often means the compute nodes are lacking resources, but that isn't the case in our environment.

Thanks in advance.