Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Failure to boot instance via volume on F20 compute node

In /var/log/nova/compute.log :-

2014-03-03 13:28:43.646 1344 WARNING nova.virt.libvirt.driver [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29b a86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] File injection into a boot from volume instance is not supported

 2014-03-03 13:30:48.024 1344 WARNING nova.virt.libvirt.volume [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 0
    2014-03-03 13:30:48.082 1344 ERROR nova.compute.manager [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Instance failed to spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1423, in _spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     block_device_info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2088, in spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     write_to_disk=True)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in to_xml
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     disk_info, rescue, block_device_info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2960, in get_guest_config
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     inst_type):
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2777, in get_guest_storage_config
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1042, in volume_driver_method
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return method(connection_info, *args, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return f(*args, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     self._run_iscsiadm(iscsi_properties, ("--rescan",))
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     check_exit_code=check_exit_code)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 21
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] 
    2014-03-03 13:30:48.678 1344 ERROR nova.virt.libvirt.driver [-] [instance: f621815f-3805-4f52-a878-9040c6a4af53] During wait destroy, instance disappeared.
    2014-03-03 13:30:48.763 1344 ERROR nova.virt.libvirt.vif [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Failed while unplugging vif
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 602, in unplug_ovs_hybrid
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     utils.execute('brctl', 'delif', br_name, v1_name, run_as_root=True)
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl delif qbr8d067af4-46 qvb8d067af4-46
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 1
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'interface qvb8d067af4-46 does not exist!\n'
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] 
    2014-03-03 13:30:50.183 1344 ERROR nova.compute.manager [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Error: Unexpected error while running command.
    Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    Exit code: 21
    Stdout: ''
    Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1043, in _build_instance
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     set_access_ip=set_access_ip)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1426, in _spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     LOG.exception(_('Instance failed to spawn'), instance=instance)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1423, in _spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     block_device_info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2088, in spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     write_to_disk=True)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in to_xml
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     disk_info, rescue, block_device_info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2960, in get_guest_config
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     inst_type):
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2777, in get_guest_storage_config
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1042, in volume_driver_method
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return method(connection_info, *args, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return f(*args, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     self._run_iscsiadm(iscsi_properties, ("--rescan",))
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     check_exit_code=check_exit_code)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 21
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]

Failure to boot instance via volume on F20 compute node

In /var/log/nova/compute.log :-

2014-03-03 13:28:43.646 1344 WARNING nova.virt.libvirt.driver [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29b a86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] File injection into a boot from volume instance is not supported

 2014-03-03 13:30:48.024 1344 WARNING nova.virt.libvirt.volume [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] ISCSI volume not yet found at: vda. Will rescan & retry.  Try number: 0
    2014-03-03 13:30:48.082 1344 ERROR nova.compute.manager [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Instance failed to spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1423, in _spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     block_device_info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2088, in spawn
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     write_to_disk=True)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in to_xml
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     disk_info, rescue, block_device_info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2960, in get_guest_config
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     inst_type):
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2777, in get_guest_storage_config
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     info)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1042, in volume_driver_method
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return method(connection_info, *args, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return f(*args, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     self._run_iscsiadm(iscsi_properties, ("--rescan",))
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     check_exit_code=check_exit_code)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 21
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:48.082 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] 
    2014-03-03 13:30:48.678 1344 ERROR nova.virt.libvirt.driver [-] [instance: f621815f-3805-4f52-a878-9040c6a4af53] During wait destroy, instance disappeared.
    2014-03-03 13:30:48.763 1344 ERROR nova.virt.libvirt.vif [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Failed while unplugging vif
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 602, in unplug_ovs_hybrid
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     utils.execute('brctl', 'delif', br_name, v1_name, run_as_root=True)
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl delif qbr8d067af4-46 qvb8d067af4-46
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 1
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'interface qvb8d067af4-46 does not exist!\n'
    2014-03-03 13:30:48.763 1344 TRACE nova.virt.libvirt.vif [instance: f621815f-3805-4f52-a878-9040c6a4af53] 
    2014-03-03 13:30:50.183 1344 ERROR nova.compute.manager [req-1bd6630e-b799-4d78-b702-f06da5f1464b df4a984ce2f24848a6b84aaa99e296f1 e896be65e94a4893b870bc29ba86d7eb] [instance: f621815f-3805-4f52-a878-9040c6a4af53] Error: Unexpected error while running command.
    Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    Exit code: 21
    Stdout: ''
    Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Traceback (most recent call last):
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1043, in _build_instance
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     set_access_ip=set_access_ip)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1426, in _spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     LOG.exception(_('Instance failed to spawn'), instance=instance)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1423, in _spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     block_device_info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2088, in spawn
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     write_to_disk=True)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3084, in to_xml
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     disk_info, rescue, block_device_info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2960, in get_guest_config
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     inst_type):
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2777, in get_guest_storage_config
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     info)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1042, in volume_driver_method
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return method(connection_info, *args, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return f(*args, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     self._run_iscsiadm(iscsi_properties, ("--rescan",))
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     check_exit_code=check_exit_code)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 177, in execute
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     return processutils.execute(*cmd, **kwargs)
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 178, in execute
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]     cmd=' '.join(cmd))
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] ProcessExecutionError: Unexpected error while running command.
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf -p 127.0.0.1:3260 --rescan
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Exit code: 21
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stdout: ''
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53] Stderr: 'iscsiadm: No session found.\n'
    2014-03-03 13:30:50.183 1344 TRACE nova.compute.manager [instance: f621815f-3805-4f52-a878-9040c6a4af53]

On Controller (Cinder host):-

[root@dallas1 ~(keystone_admin)]$ tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2010-10.org.openstack:volume-061e4f6f-260a-41fa-9cb0-d747682e4d92
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 5369 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder-volumes/volume-061e4f6f-260a-41fa-9cb0-d747682e4d92
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 2: iqn.2010-10.org.openstack:volume-36f3fa59-c35f-4fe1-98c0-7f2813862107
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00020000
            SCSI SN: beaf20
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00020001
            SCSI SN: beaf21
            Size: 7516 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder-volumes/volume-36f3fa59-c35f-4fe1-98c0-7f2813862107
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 3: iqn.2010-10.org.openstack:volume-4704ead1-c17b-405f-b960-e65eab5146e3
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00030000
            SCSI SN: beaf30
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00030001
            SCSI SN: beaf31
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder-volumes/volume-4704ead1-c17b-405f-b960-e65eab5146e3
            Backing store flags: 
    Account information:
    ACL information:
        ALL
Target 4: iqn.2010-10.org.openstack:volume-900db960-e1e6-4dc9-9f97-0ee2298815bf
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00040000
            SCSI SN: beaf40
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00040001
            SCSI SN: beaf41
            Size: 6442 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /dev/cinder-volumes/volume-900db960-e1e6-4dc9-9f97-0ee2298815bf
            Backing store flags: 
    Account information:
    ACL information:
        ALL