Ask Your Question
6

What causes Metadata service to be very slow?

asked 2013-07-02 04:35:04 -0500

Davide gravatar image

updated 2013-07-05 08:07:01 -0500

smaffulli gravatar image

Hello,

on a fresh Grizzly deployment, our metadata service is very slow :

$ time curl http://169.254.169.254/2009-04-04/meta-data/instance-id
i-0000000d
real    0m16.295s
user    0m0.012s
sys 0m0.000s

This causes a very long boot time for instances (like 4-5 minutes).

At the moment we are not injecting any file or script, neither we are using cloud-init. Nothing in the logs seems to explain this issue.

What could be the cause of such long delays?

Thanks in advance, Cheers

edit retag flag offensive close merge delete

Comments

I'm seeing this as well, except most of the metadata requests come back in 0.3 secs, whereas 3 or 4 of all the ones made by a single boot from a single instance take as long as the above (~8-17 secs). Also noticed that the quicker ones usually only hit once every ~2 seconds, rather than rapid-fire.

briancline gravatar imagebriancline ( 2013-07-24 16:32:01 -0500 )edit

As a clarification, I *am* using cloud-init in my environment.

briancline gravatar imagebriancline ( 2013-07-24 16:33:47 -0500 )edit

I am seeing the same with a fresh grizzly install. It seems to make no difference whether or not cloud-init is actually being used on the instance in question.

tms gravatar imagetms ( 2013-07-29 20:14:28 -0500 )edit

Finally i've abandoned the ec2 metadata and configured the openstack config disk in nova. This solves any slowness at boot. EC2 metadata is still slow, but it's not a problem anymore. Also when using cloud-init data at boot, ec2 metadata becomes so slow that you can wait till 12-13 minutes before booting. So bye-bye EC2, welcome config disk!

Davide gravatar imageDavide ( 2013-08-14 06:42:06 -0500 )edit

Same problem here. Once I removed ec2 from the enabled apis, everything went fine.

javsalgar gravatar imagejavsalgar ( 2013-10-19 09:46:20 -0500 )edit

3 answers

Sort by ยป oldest newest most voted
1

answered 2013-11-14 20:29:54 -0500

tscopp gravatar image

I experimented with various fixes for this (up to and including patchsets that fix what I believe the problem to be) but I think they got to it somewhere in Havana because it's no longer an issue. If there's interest I'd be happy to share my changes with anyone locked in to running a Grizzly cloud long term :)

edit flag offensive delete link more

Comments

I'm running Openstack Havana, i'd be interested in some details about your fixes. My current avg time is around 15-20s for a 1 controller, 6x24cores compute node stack with ~50 instances

Pierre gravatar imagePierre ( 2013-11-19 21:28:25 -0500 )edit
3

answered 2013-11-13 14:46:38 -0500

ShillaSaebi gravatar image

So it looks like once you abandon the EC2 metadata and configure the OpenStack config disk in nova, the problem is resolved.

edit flag offensive delete link more
0

answered 2015-03-18 23:52:51 -0500

updated 2015-03-18 23:53:13 -0500

Probably just slow crawling ec2 metadata service. Not sure why yet... image description

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2013-07-02 04:35:04 -0500

Seen: 2,168 times

Last updated: Mar 18 '15