Openstack (Train )instance not launching

asked 2020-04-10 04:35:45 -0500

Vinstack gravatar image

Details : Openstack (Train )instance not launching, Failed to copy image to volume: qemu-img: Could not open '/var/lib/cinder/conversion/tmpHPa4Zkopenstack@lvm': Image is not in qcow2 format

Hi All, Openstack instance launch always fails with the following error: This error is occurring even for iso , img images . Tried an image to convert to qcow2 no luck. Can anyone please help its days that I cant get this instances launched and it fails every time with ERROR

2020-04-10 00:35:34.181 250692 WARNING cinder.volume.manager [req-d3a5dc5d-5551-41e7-97db-a5040efc6945 6ad05a7481174e2bb7b5e309bd40d341 d30e18aecf034366be85e9e5429fe1bf - default default] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (985381a6-3e25-411a-bb30-91a7857d6ebf) transitioned into state 'REVERTED' from state 'REVERTING' 2020-04-10 00:35:34.184 250692 WARNING cinder.volume.manager [req-d3a5dc5d-5551-41e7-97db-a5040efc6945 6ad05a7481174e2bb7b5e309bd40d341 d30e18aecf034366be85e9e5429fe1bf - default default] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (48f89f5b-558a-4332-9030-de09640b15fe) transitioned into state 'REVERTED' from state 'REVERTING' 2020-04-10 00:35:34.186 250692 WARNING cinder.volume.manager [req-d3a5dc5d-5551-41e7-97db-a5040efc6945 6ad05a7481174e2bb7b5e309bd40d341 d30e18aecf034366be85e9e5429fe1bf - default default] Flow 'volume_create_manager' (32a45c1b-cec0-449f-9518-c94cc5f0ddf0) transitioned into state 'REVERTED' from state 'RUNNING' 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server [req-d3a5dc5d-5551-41e7-97db-a5040efc6945 6ad05a7481174e2bb7b5e309bd40d341 d30e18aecf034366be85e9e5429fe1bf - default default] Exception during message handling: ImageCopyFailure: Failed to copy image to volume: qemu-img: Could not open '/var/lib/cinder/conversion/tmpHPa4Zkopenstack@lvm': Image is not in qcow2 format 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server result = func(ctxt, new_args) 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 212, in wrapper 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server result = f(args, *kwargs) 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 749, in create_volume 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server _run_flow() 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 741, in _run_flow 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server flow_engine.run() 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2020-04-10 00:35:34.190 250692 ERROR oslo_messaging.rpc ... (more)

edit retag flag offensive close merge delete

Comments

I have tried with multiple images and qcow2 conversions, No Luck!

Vinstack gravatar imageVinstack ( 2020-04-10 04:37:14 -0500 )edit

Wild guess: Perhaps the disk space in /var/lib/cinder is insufficient?

I would also check the Cinder Volume and the Glance API logs.

Is the file /var/lib/cinder/conversion/tmp.... still there?

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-10 05:23:13 -0500 )edit

Hi Bernd, Thanks for your replies ! There is not tmp folder under conversation I tried to clear the space under /var/lib/cinder which contained all the deleted volumes list.....and accidentally I deleted the "rm: remove regular file ‘cinder-volumes’? y". Is this fine?

Vinstack gravatar imageVinstack ( 2020-04-10 06:51:31 -0500 )edit

Saved logs here https://drive.google.com/drive/folders/1NU4vuCkrUSGjYRM3Y7h4Y2owTPv-opdb (https://drive.google.com/drive/folder...)

Vinstack gravatar imageVinstack ( 2020-04-10 06:56:08 -0500 )edit

I don't know what the file cinder-volumes is/was for, but it doesn't sound good. What OpenStack flavour are you running?

To check storage space, use the df command. Don't remove files whose purpose you don't know.

I will have a look at the logs after a good night's sleep.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-10 10:12:29 -0500 )edit

Hi Bernd. I am using Openstack Train

Flavour. m1.medium VCPU's 2 RAM: 4 GB Total DISK: 40 GB RootDisk:40 GB Ephemeral Disk: 0 GB Public Yes

df has lot of free space

I have attached nova, neutron, cinder, glace logs for your reference. GoodNight and Thanks for looking into and your kind help

Vinstack gravatar imageVinstack ( 2020-04-10 11:27:58 -0500 )edit

Perhaps you have plenty of disk space now, after removing cinder-volumes. However, I do see repeated errors in the Glance API log related to database connections. My guess: Cinder is unable to get the full image from Glance, or perhaps even no image data at all.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-10 21:11:04 -0500 )edit

I would research the "Too many connections" error. I believe that there is a configuration setting that addresses it.

By the way, I wanted to know how you deployed the cloud. cinder-volumes may contain the physical volume used to implement Cinder's LVM volume group. If so, you destroyed it.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-10 21:12:55 -0500 )edit

Hi Bernd, I have deployed openstack using packstack installation.

[root@openstack ~]# vgs VG #PV #LV #SN Attr VSize VFree centos 2 3 0 wz--n- 1.09t 0 cinder-volumes 2 2 0 wz--n- 70.59g 50.98g [root@openstack ~]#

Cinder volume available and I/O's work

Vinstack gravatar imageVinstack ( 2020-04-11 02:42:17 -0500 )edit

not able to paste the pvs results: /dev/loop4 cinder-volumes lvm2 a-- <50.00g <50.00g /dev/sdb2 centos lvm2 a-- <557.91g 0 /dev/sdc1 centos lvm2 a-- <558.91g 0 /dev/sdf1 lvm2 --- <111.79g <111.79g

Vinstack gravatar imageVinstack ( 2020-04-11 02:43:50 -0500 )edit

There are two major issues as of i see: Cinder volumes gets created on some launch of instances but when spawning the instance it fails with port binding issue that can be seen in nova and neutron logs I am confused...

Vinstack gravatar imageVinstack ( 2020-04-11 03:24:32 -0500 )edit

That is different from the above description.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 04:34:33 -0500 )edit

Hi Bernd, The original issue is I am not able to launch the instance. If you see in cinder logs the volume creates are successful in some cases and I have mentioned about the port binding as well in the issue

Could you please help check

Vinstack gravatar imageVinstack ( 2020-04-11 08:01:06 -0500 )edit

I told you about the Glance problem. I don't know more about these DB connection issues than you. While you research that, I will check the Nova and Neutron logs for messages related to port binding.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 09:39:59 -0500 )edit

Thanks for your kind help, Bernd. Fixing other parts as per your suggestion.

Vinstack gravatar imageVinstack ( 2020-04-11 10:12:17 -0500 )edit

I suggest you have a closer look at the logs. For example, the same DB connection error is in the DHCP agent log.

The Openvswitch Agent log repeats "Invalid tunnel type specified: flat", which might well be due to misconfiguration.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 21:49:29 -0500 )edit

Neutron's server log contains the bind failure messages. A detail: 'physical_network': None,. Again, this indicates a misconfiguration in my opinion, though I don't know if that is normal for Geneve networks.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 21:52:20 -0500 )edit

One detail confuses me (it might be irrelevant): The Packstack instructions are for Stein, but you say you installed Train? Which instructions did you follow? Can you share your answer file (or perhaps you use no answer file?)?

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 21:54:16 -0500 )edit

Regarding the tunnel type: It is defined in the agent section of the openvswitch configuration file.

Bernd Bausch gravatar imageBernd Bausch ( 2020-04-11 21:57:35 -0500 )edit

Hi Bernd....I have used packstack - all in one installation with no answer file. I am rechecking with the "openvswitch configuration file" and revert.

Vinstack gravatar imageVinstack ( 2020-04-13 00:27:59 -0500 )edit