What causes Metadata service to be very slow?

asked 2013-07-02 04:35:04 -0600

Davide
81 ●4 ●4 ●8

updated 2013-07-05 08:07:01 -0600

smaffulli
6981 ●38 ●68 ●102 http://maffulli.net/

Hello,

on a fresh Grizzly deployment, our metadata service is very slow :

$ time curl http://169.254.169.254/2009-04-04/meta-data/instance-id
i-0000000d
real    0m16.295s
user    0m0.012s
sys 0m0.000s

This causes a very long boot time for instances (like 4-5 minutes).

At the moment we are not injecting any file or script, neither we are using cloud-init. Nothing in the logs seems to explain this issue.

What could be the cause of such long delays?

Thanks in advance, Cheers

edit retag flag offensive close merge delete

Comments

I'm seeing this as well, except most of the metadata requests come back in 0.3 secs, whereas 3 or 4 of all the ones made by a single boot from a single instance take as long as the above (~8-17 secs). Also noticed that the quicker ones usually only hit once every ~2 seconds, rather than rapid-fire.

briancline ( 2013-07-24 16:32:01 -0600 )edit

As a clarification, I *am* using cloud-init in my environment.

briancline ( 2013-07-24 16:33:47 -0600 )edit

I am seeing the same with a fresh grizzly install. It seems to make no difference whether or not cloud-init is actually being used on the instance in question.

tms ( 2013-07-29 20:14:28 -0600 )edit

Finally i've abandoned the ec2 metadata and configured the openstack config disk in nova. This solves any slowness at boot. EC2 metadata is still slow, but it's not a problem anymore. Also when using cloud-init data at boot, ec2 metadata becomes so slow that you can wait till 12-13 minutes before booting. So bye-bye EC2, welcome config disk!

Davide ( 2013-08-14 06:42:06 -0600 )edit

Same problem here. Once I removed ec2 from the enabled apis, everything went fine.

javsalgar ( 2013-10-19 09:46:20 -0600 )edit

add a comment

1

answered 2013-11-14 20:29:54 -0600

tscopp
26 ●1 ●1

I experimented with various fixes for this (up to and including patchsets that fix what I believe the problem to be) but I think they got to it somewhere in Havana because it's no longer an issue. If there's interest I'd be happy to share my changes with anyone locked in to running a Grizzly cloud long term :)

edit flag offensive delete link

Comments

I'm running Openstack Havana, i'd be interested in some details about your fixes. My current avg time is around 15-20s for a 1 controller, 6x24cores compute node stack with ~50 instances