can't start an instance after manually detaching volume, because is still looking for it

asked 2020-08-28 01:52:32 -0500

sacha-m gravatar image

updated 2020-08-28 10:26:22 -0500

I have an instance that had some problems with a volume. I detached the volumes with: cinder reset-state --attach-status detached [ID_Volume] and then delete it: cinder delete [ID_Volume] Now the instance is in error state, so I reset ir to active, so I try to start it with a hard reboot, but it's still trying to find the missing volume:

2020-08-28 02:30:19.886 6 INFO nova.compute.manager [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] [instance: 3b86e532-176a-4ee7-bfd4-cf625191911b] Rebooting instance
2020-08-28 02:30:21.052 6 WARNING nova.compute.manager [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] [instance: 3b86e532-176a-4ee7-bfd4-cf625191911b] trying to reboot a non-running instance: (state: 0 expected: 1)
2020-08-28 02:30:21.237 6 INFO nova.virt.libvirt.driver [-] [instance: 3b86e532-176a-4ee7-bfd4-cf625191911b] Instance destroyed successfully.
2020-08-28 02:30:21.312 6 INFO os_vif [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:23:ba:73,bridge_name='qbra00218a5-69',has_traffic_filtering=True,id=a00218a5-69ca-48ee-953f-2fa6d368a192,network=Network(1440cf9e-52e6-4917-88bf-975d7f630e87),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapa00218a5-69')
2020-08-28 02:30:21.365 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Couldn't find iSCSI nodes because iscsiadm err: iscsiadm: No records found
2020-08-28 02:30:21.405 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
2020-08-28 02:30:21.451 6 INFO os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Trying to connect to iSCSI portal 172.16.230.2:3260
2020-08-28 02:30:21.735 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
2020-08-28 02:30:22.799 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-b0be29ca-dbfd-4aed-aaf1-8fde34c93ed8 on portal 172.16.230.2:3260 (exit code 19).: ProcessExecutionError: Unexpected error while running command.
2020-08-28 02:30:22.800 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Failed to connect to iSCSI portal 172.16.230.2:3260.
2020-08-28 02:30:22.893 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
2020-08-28 02:30:25.018 6 INFO os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Trying to connect to iSCSI portal 172.16.230.2:3260
2020-08-28 02:30:25.352 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] iscsiadm stderr output when getting sessions: iscsiadm: No active sessions.
: VolumeDeviceNotFound: Volume device not found at .
2020-08-28 02:30:26.409 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Failed to login iSCSI target iqn.2010-10.org.openstack:volume-b0be29ca-dbfd-4aed-aaf1-8fde34c93ed8 on portal 172.16.230.2:3260 (exit code 19).: ProcessExecutionError: Unexpected error while running command.
2020-08-28 02:30:26.410 6 WARNING os_brick.initiator.connectors.iscsi [req-2bdbd99b-bfce-47b1-9b20-c3d909171a9e dfff51be035147dd9a0d60f6ad01312f f590b293a0ca4cf8b209c6bff7160811 - default default] Failed to connect to iSCSI portal 172.16.230.2:3260.: VolumeDeviceNotFound: Volume device not ...
(more)
edit retag flag offensive close merge delete

Comments

It's hard to tell what your actions may have caused. Do you have access to the database? You could take a look into the nova database, table block_device_mapping and paste the output here. Are you sure deleting the detached volume caused this or was there a problem beforehand? Are all services up?

eblock gravatar imageeblock ( 2020-08-28 02:31:10 -0500 )edit

All servers are up, and I'm not 100% sure that the deleting volume cause it. Here is the table you requested... but it seems I have to find out how to make KVM to put the instance 3b86e532-176a-4ee7-bfd4-cf625191911b into its inventory or something like that (look at the update 2 i just added)

sacha-m gravatar imagesacha-m ( 2020-08-28 10:25:05 -0500 )edit