Ask Your Question
0

Glance - Client disconnected before sending all data to backend

asked 2014-05-13 02:29:17 -0500

updated 2014-05-14 21:47:06 -0500

SamYaple gravatar image

I'm facing again the same issue as this post.

Here my glance api log,

I'm uploading a 30G vmdk disk image. I do not think it is a problem of space in nfs datastore (36% used, available space 370GB).

Here the output of df command:

root@havana-controller:/var/log/glance# df
Filesystem                      1K-blocks      Used Available Use% Mounted on
/dev/sda1                        59862140  40482468  16338856  72% /
udev                              2016104         4   2016100   1% /dev
tmpfs                              404984       252    404732   1% /run
none                                 5120         0      5120   0% /run/lock
none                              2024920         0   2024920   0% /run/shm
x.x.x.x:/openstack/glance 597688320 209738624 387949696  36% /var/lib/glance/images

Any hint on possible solutions will be appreciated. Thanks in advance.

UPDATE

I tried copying the 30GB vmdk file on /var/lib/glance/images/ and it perfectly worked:

root@havana-controller:/var/lib/glance/images# ls -ltrh
total 224G
-rw-r----- 1 glance glance 4.9G Feb 11 09:46 0f312efb-d4cd-459c-b796-30a14df987c0
-rw-r----- 1 glance glance 2.0G Feb 11 09:48 2b478ffa-7457-4d4d-96fb-be55fa728364
-rw-r----- 1 glance glance 4.9G Feb 11 09:51 5432b526-4e01-41a0-92ca-e58a1b2450bf
-rw-r----- 1 glance glance 4.8G Feb 11 09:56 a3b4194d-181c-497c-bb32-df9afedc1fe9
-rw-r----- 1 glance glance 1.9G Feb 11 09:59 ac50769a-6df6-4883-818a-904392424a66
-rw-r----- 1 glance glance  13M Feb 11 09:59 b22e7d20-1c20-43a4-b403-357e95a0c9e8
-rw-r----- 1 glance glance 1.7G Feb 11 10:01 ef834389-ad08-47cd-865b-7c1951f29eea
-rw-r--r-- 1 glance glance  151 Feb 11 10:03 README.txt
-rw-r----- 1 glance glance  20G Feb 20 14:33 292efdf7-f244-4f5d-9e08-6c055c85465e
-rw-r----- 1 glance glance  40G Feb 27 19:14 6223273f-8768-4d93-860e-ea413e599ca6
-rw-r----- 1 glance glance  60G Mar 18 14:53 2432d887-ec79-49ca-b01f-3078f3875fcd
-rw-r----- 1 glance glance  20G Apr  4 12:07 125f6843-1852-4193-9868-a9e2de61b581
-rw-r----- 1 glance glance  30G Apr 16 11:47 c9a3b842-236a-4236-8742-1570abcc4453
-rw-r----- 1 glance glance 3.7G Apr 25 16:44 ecd0d909-92ca-4559-b5b9-937ec450997f
-rwxr-xr-x 1 root   root    30G May 14  2014 infse-cygwin-flat.vmdk

So I could exclude any issue related on running out of space on the NFS share.

Thanks in advance for any suggestion, Antonio

edit retag flag offensive close merge delete

Comments

1

Just so you are aware you posted your glance mysql public address and password, I have removed that conf file link from your post. Feel free to revert my edit if you so wish.

SamYaple gravatar imageSamYaple ( 2014-05-14 21:48:49 -0500 )edit

I see my answer was marked "Solved", was this by you? If you did solve the problem, can you give me an update on the solution? I am very intrested in how this turned out.

SamYaple gravatar imageSamYaple ( 2014-05-23 07:45:58 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
1

answered 2014-05-14 22:00:34 -0500

SamYaple gravatar image

I do remember this issue from previously. I had asked around at this time. No one that I spoke with had seen this before (when space _wasn't_ the issue).

This issue seems to have the same symptoms, though they talk about running glance behind apache WSGI, which I don't think you are doing. Nevertheless, you might think about trying out that while loop instead. It does talk about being near the last chunk of the file that it fails on, just like you.

My only new suggestion would be trying v2 api for glance and seeing if that resolves it for you.

Also opening up a bug for the issue. If someone can reproduce it, it can get fixed.

edit flag offensive delete link more

Comments

The problem happens only with NetApp FAS. The issue were not related for sure with space on nfs share.

Our workaround consists on switching on standard NFS exposed by a linux machine, and the issue does not appear anymore.

As soon as we get a better comprehension of the phenomenon we will post the right solution here.

Thanks for your help!

Antonio G. gravatar imageAntonio G. ( 2014-05-23 08:01:44 -0500 )edit

Ah yes. The NetApp. It has never played well with others in my experience. I would love to see the issue get resolved because as Openstack gets more widely adopted you can be assured you wont be the only one to experince that issue.

SamYaple gravatar imageSamYaple ( 2014-05-23 08:10:25 -0500 )edit
0

answered 2015-03-10 17:48:34 -0500

I am hitting this as well. Can you check whether restart of glance does help? I think it may be some auth timeout issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1133566 (https://bugzilla.redhat.com/show_bug....) https://bugs.launchpad.net/glance-store/+bug/1361197 (https://bugs.launchpad.net/glance-sto...) https://bugs.launchpad.net/nova/+bug/1240355 (https://bugs.launchpad.net/nova/+bug/...)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-05-13 02:29:17 -0500

Seen: 599 times

Last updated: May 14 '14