vm not starting, missing volume?

asked 2018-10-08 00:59:55 -0600

masber gravatar image

Dear Openstack community,

I had to reboot my server and now I can’t start one of the vms (danrod-server) which boots from volume.

vm details

[root@openstack ~(keystone_admin)]# openstack server show danrod-server
+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                   |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | openstack.localdomain                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | openstack.localdomain                                    |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000069                                        |
| OS-EXT-STS:power_state               | NOSTATE                                                  |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | stopped                                                  |
| OS-SRV-USG:launched_at               | 2018-05-29T11:22:17.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | privatenetwork=192.168.1.105, 129.94.14.238              |
| config_drive                         |                                                          |
| created                              | 2018-05-29T11:22:09Z                                     |
| flavor                               | xlarge.cpu.xxxlarge.ram (6)                              |
| hostId                               | ecef276660cd714fe626073a18c11fe1c00bec91c15516178fb6ac28 |
| id                                   | 1e59f329-072e-48ae-abf1-266eba437508                     |
| image                                |                                                          |
| key_name                             | None                                                     |
| name                                 | danrod-server                                            |
| os-extended-volumes:volumes_attached | [{u'id': u'f1ac2e94-b0ed-4089-898f-5b6467fb51e3'}]       |
| project_id                           | d58cf22d960e4de49b71658aee642e94                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'admin'}, {u'name': u'R-Studio Server'}]     |
| status                               | SHUTOFF                                                  |
| updated                              | 2018-10-08T02:52:41Z                                     |
| user_id                              | c412f34c353244eabecd4b6dc4d36392                         |
+--------------------------------------+----------------------------------------------------------+

List of volumes

[root@openstack ~(keystone_admin)]# openstack volume list --all
+--------------------------------------+--------------+--------+------+----------------------------------------+
| ID                                   | Display Name | Status | Size | Attached to                            |
+--------------------------------------+--------------+--------+------+----------------------------------------+
| f1ac2e94-b0ed-4089-898f-5b6467fb51e3 |              | in-use |  700 | Attached to danrod-server on /dev/vda  |
+--------------------------------------+--------------+--------+------+----------------------------------------+

nova-compute.log

2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] Multipath discovery for iSCSI not enabled.
2018-10-08 13:51:46.476 4015 INFO os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] Trying to connect to iSCSI portal 129.94.14.254:3260
2018-10-08 13:51:46.504 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 0.
2018-10-08 13:51:47.510 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 1.
2018-10-08 13:51:51.519 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 2.
2018-10-08 13:52:00.527 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 3.
2018-10-08 13:52:16.535 4015 WARNING os_brick.initiator.connectors.iscsi [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] ISCSI volume not yet found at: [u'/dev/disk/by-path/ip-129.94.14.254:3260-iscsi-iqn.2010-10.org.openstack:volume-f1ac2e94-b0ed-4089-898f-5b6467fb51e3-lun-0']. Will rescan & retry.  Try number: 4.
2018-10-08 13:52:23.357 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Auditing locally available compute resources for node openstack.localdomain
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Total usable vcpus: 56, total allocated vcpus: 52
2018-10-08 13:52:23.713 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Final resource view: name=openstack.localdomain phys_ram=524173MB used_ram=454608MB phys_disk=9312GB used_disk=1540GB total_vcpus=56 used_vcpus=52 pci_stats=[]
2018-10-08 13:52:23.733 4015 INFO nova.compute.resource_tracker [req-359039dc-29dc-4f06-805f-0ca943bb600f - - - - -] Compute_service record updated for openstack.localdomain:openstack.localdomain
2018-10-08 13:52:41.665 4015 INFO nova.compute.manager [req-d9ed864b-9a4f-4182-b91b-9a2f1e38fb02 c412f34c353244eabecd4b6dc4d36392 d58cf22d960e4de49b71658aee642e94 - - -] [instance: 1e59f329-072e-48ae-abf1-266eba437508 ...
(more)
edit retag flag offensive close merge delete

Comments

Hi

As I can see the iscsi target for affected volume is no more reachable from the compute hosting the affected VM. Please check the connectivity from the hosting compute towards the iscsi target address i.e 129.94.14.254:3260

BR//Kamal

ekalder_88 gravatar imageekalder_88 ( 2018-10-08 02:26:16 -0600 )edit

Cinder complains about not reaching APMQ at 08:52:58. The Nova errors are around 13:52:00. So, either the Cinder errors are not related (might be during startup or so), or you need to ensure that the Cinder and the Compute servers have the same time.

Bernd Bausch gravatar imageBernd Bausch ( 2018-10-09 04:57:35 -0600 )edit