Glance - Client disconnected before sending all data to backend

asked 2014-05-13 02:29:17 -0600

Antonio G.

1788 ●17 ●27 ●45 http://www.antoniogian...

updated 2014-05-14 21:47:06 -0600

SamYaple

2526 ●7 ●21 ●45

I'm facing again the same issue as this post.

Here my glance api log,

I'm uploading a 30G vmdk disk image. I do not think it is a problem of space in nfs datastore (36% used, available space 370GB).

Here the output of df command:

root@havana-controller:/var/log/glance# df
Filesystem                      1K-blocks      Used Available Use% Mounted on
/dev/sda1                        59862140  40482468  16338856  72% /
udev                              2016104         4   2016100   1% /dev
tmpfs                              404984       252    404732   1% /run
none                                 5120         0      5120   0% /run/lock
none                              2024920         0   2024920   0% /run/shm
x.x.x.x:/openstack/glance 597688320 209738624 387949696  36% /var/lib/glance/images

Any hint on possible solutions will be appreciated. Thanks in advance.

UPDATE

I tried copying the 30GB vmdk file on /var/lib/glance/images/ and it perfectly worked:

root@havana-controller:/var/lib/glance/images# ls -ltrh
total 224G
-rw-r----- 1 glance glance 4.9G Feb 11 09:46 0f312efb-d4cd-459c-b796-30a14df987c0
-rw-r----- 1 glance glance 2.0G Feb 11 09:48 2b478ffa-7457-4d4d-96fb-be55fa728364
-rw-r----- 1 glance glance 4.9G Feb 11 09:51 5432b526-4e01-41a0-92ca-e58a1b2450bf
-rw-r----- 1 glance glance 4.8G Feb 11 09:56 a3b4194d-181c-497c-bb32-df9afedc1fe9
-rw-r----- 1 glance glance 1.9G Feb 11 09:59 ac50769a-6df6-4883-818a-904392424a66
-rw-r----- 1 glance glance  13M Feb 11 09:59 b22e7d20-1c20-43a4-b403-357e95a0c9e8
-rw-r----- 1 glance glance 1.7G Feb 11 10:01 ef834389-ad08-47cd-865b-7c1951f29eea
-rw-r--r-- 1 glance glance  151 Feb 11 10:03 README.txt
-rw-r----- 1 glance glance  20G Feb 20 14:33 292efdf7-f244-4f5d-9e08-6c055c85465e
-rw-r----- 1 glance glance  40G Feb 27 19:14 6223273f-8768-4d93-860e-ea413e599ca6
-rw-r----- 1 glance glance  60G Mar 18 14:53 2432d887-ec79-49ca-b01f-3078f3875fcd
-rw-r----- 1 glance glance  20G Apr  4 12:07 125f6843-1852-4193-9868-a9e2de61b581
-rw-r----- 1 glance glance  30G Apr 16 11:47 c9a3b842-236a-4236-8742-1570abcc4453
-rw-r----- 1 glance glance 3.7G Apr 25 16:44 ecd0d909-92ca-4559-b5b9-937ec450997f
-rwxr-xr-x 1 root   root    30G May 14  2014 infse-cygwin-flat.vmdk

So I could exclude any issue related on running out of space on the NFS share.

Thanks in advance for any suggestion, Antonio

edit retag flag offensive close merge delete

Comments

Just so you are aware you posted your glance mysql public address and password, I have removed that conf file link from your post. Feel free to revert my edit if you so wish.

SamYaple ( 2014-05-14 21:48:49 -0600 )edit

I see my answer was marked "Solved", was this by you? If you did solve the problem, can you give me an update on the solution? I am very intrested in how this turned out.

SamYaple ( 2014-05-23 07:45:58 -0600 )edit

add a comment

2 answers

Sort by » oldest newest most voted

answered 2014-05-14 22:00:34 -0600

SamYaple

2526 ●7 ●21 ●45

I do remember this issue from previously. I had asked around at this time. No one that I spoke with had seen this before (when space _wasn't_ the issue).

This issue seems to have the same symptoms, though they talk about running glance behind apache WSGI, which I don't think you are doing. Nevertheless, you might think about trying out that while loop instead. It does talk about being near the last chunk of the file that it fails on, just like you.

My only new suggestion would be trying v2 api for glance and seeing if that resolves it for you.

Also opening up a bug for the issue. If someone can reproduce it, it can get fixed.

edit flag offensive delete link

Comments

The problem happens only with NetApp FAS. The issue were not related for sure with space on nfs share.

Our workaround consists on switching on standard NFS exposed by a linux machine, and the issue does not appear anymore.

As soon as we get a better comprehension of the phenomenon we will post the right solution here.

Thanks for your help!

Antonio G. ( 2014-05-23 08:01:44 -0600 )edit

Ah yes. The NetApp. It has never played well with others in my experience. I would love to see the issue get resolved because as Openstack gets more widely adopted you can be assured you wont be the only one to experince that issue.

SamYaple ( 2014-05-23 08:10:25 -0600 )edit

add a comment

answered 2015-03-10 17:48:34 -0600

Jaroslav Henner
1

I am hitting this as well. Can you check whether restart of glance does help? I think it may be some auth timeout issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1133566 (https://bugzilla.redhat.com/show_bug....) https://bugs.launchpad.net/glance-store/+bug/1361197 (https://bugs.launchpad.net/glance-sto...) https://bugs.launchpad.net/nova/+bug/1240355 (https://bugs.launchpad.net/nova/+bug/...)

edit flag offensive delete link

add a comment

Glance - Client disconnected before sending all data to backend

Comments

2 answers

Comments

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Glance - Client disconnected before sending all data to backend edit

Comments

2 answers

Comments

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Glance - Client disconnected before sending all data to backend