Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Why can I not boot an instance when using ceph?

I cannot boot an instance when using ceph.

I see the following in nova-compute.log

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [req-ae75f8b0-6c67-4829-86cc-3d07fb97ded1 ff7938552f984bcc865648df072bbbcd 6ce207968e2b46a6b2a485f329affedb - default default] [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] Instance failed to spawn: libvirtError: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] Traceback (most recent call last):

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _build_resources

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     yield resources

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2026, in _build_and_run_instance

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     block_device_info=block_device_info)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2829, in spawn

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     destroy_disks_on_failure=True)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5302, in _create_domain_and_network

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     destroy_disks_on_failure)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self.force_reraise()

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(self.type_, self.value, self.tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5272, in _create_domain_and_network

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     post_xml_callback=post_xml_callback)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5190, in _create_domain

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     guest.launch(pause=pause)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self._encoded_xml, errors='ignore')

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self.force_reraise()

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(self.type_, self.value, self.tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     return self._domain.createWithFlags(flags)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     result = proxy_call(self._autowrap, f, *args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     rv = execute(f, *args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(c, e, tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     rv = meth(*args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] libvirtError: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]

and nova-conductor.log.

2017-10-25 11:44:50.179 21 ERROR nova.scheduler.utils [req-3e3b50ba-5f5f-4aa0-a285-2a4403f34c02 ff7938552f984bcc865648df072bbbcd 6ce207968e2b46a6b2a485f329affedb - default default] [instance: c3867508-e0b0-4b1c-a489-daa5a27665d0] Error from last host: compute01 (node compute01): [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1875, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2105, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance c3867508-e0b0-4b1c-a489-daa5a27665d0 was re-scheduled: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'\n"]

Why can I not boot an instance when using ceph?

I cannot boot an instance when using ceph.ceph with Oracle VM server release 3.4.4 compute host.

I see the following in nova-compute.log

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [req-ae75f8b0-6c67-4829-86cc-3d07fb97ded1 ff7938552f984bcc865648df072bbbcd 6ce207968e2b46a6b2a485f329affedb - default default] [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] Instance failed to spawn: libvirtError: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] Traceback (most recent call last):

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2211, in _build_resources

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     yield resources

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2026, in _build_and_run_instance

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     block_device_info=block_device_info)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2829, in spawn

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     destroy_disks_on_failure=True)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5302, in _create_domain_and_network

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     destroy_disks_on_failure)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self.force_reraise()

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(self.type_, self.value, self.tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5272, in _create_domain_and_network

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     post_xml_callback=post_xml_callback)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5190, in _create_domain

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     guest.launch(pause=pause)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self._encoded_xml, errors='ignore')

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     self.force_reraise()

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(self.type_, self.value, self.tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     return self._domain.createWithFlags(flags)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     result = proxy_call(self._autowrap, f, *args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     rv = execute(f, *args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     six.reraise(c, e, tb)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     rv = meth(*args, **kwargs)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574] libvirtError: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'

2017-10-25 11:24:11.700 1 ERROR nova.compute.manager [instance: 21c2742d-d2f7-49ff-8687-39e264b6d574]

and nova-conductor.log.

2017-10-25 11:44:50.179 21 ERROR nova.scheduler.utils [req-3e3b50ba-5f5f-4aa0-a285-2a4403f34c02 ff7938552f984bcc865648df072bbbcd 6ce207968e2b46a6b2a485f329affedb - default default] [instance: c3867508-e0b0-4b1c-a489-daa5a27665d0] Error from last host: compute01 (node compute01): [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1875, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2105, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance c3867508-e0b0-4b1c-a489-daa5a27665d0 was re-scheduled: Secret not found: rbd no secret matches uuid '7596eb15-cb51-403d-8a4b-3808131686a2'\n"]