euca2ools returns HTTP/1.1 500 Internal Server Error

asked 2013-10-28 00:47:24 -0600

LDC gravatar image

updated 2014-01-22 15:12:56 -0600

Evgeny gravatar image


I have an Openstack deployment in HA, with 2 compute nodes and 3 controllers. Everything is working fine, however as soon as I try using EC2 Compatibility APIs though euca2ools commands I am always getting HTTP 500 error. nova*.log do not show anything that could help me pointing out where the problem could be. Native openstack calls works properly. I haven't find much documentation on how troubleshooting this issue and I would appreciate if anybody can provide some hints, ideas.

Here few information gathered from the controllers

1) ec2 credential in keystone ec-credentials-list --user-id match with the EC2 Access and Secret keys. EC2_AUTH is not set as not not defined in my I did not have to create keystone credentials as many tutorial describe as it seems they are created when downloading the .zip file

2) api-paste.ini is the same in the 3 controllers, I paste here the config for ec2 and ec2cloud

# EC2 #

use = egg:Paste#urlmap
/services/Cloud: ec2cloud

use = call:nova.api.auth:pipeline_factory
noauth = ec2faultwrap logrequest ec2noauth cloudrequest validator ec2executor
keystone = ec2faultwrap logrequest ec2keystoneauth cloudrequest validator ec2executor

paste.filter_factory = nova.api.ec2:FaultWrapper.factory

paste.filter_factory = nova.api.ec2:RequestLogging.factory

paste.filter_factory = nova.api.ec2:Lockout.factory

paste.filter_factory = nova.api.ec2:EC2KeystoneAuth.factory

paste.filter_factory = nova.api.ec2:NoAuth.factory

controller =
paste.filter_factory = nova.api.ec2:Requestify.factory

paste.filter_factory = nova.api.ec2:Authorizer.factory

paste.filter_factory = nova.api.ec2:Validator.factory

paste.app_factory = nova.api.ec2:Executor.factory

3) In /etc/nova/nova.conf enabled_apis include ec2, however in my nova.conf ec2_listen_port=8873 is not present but only ec2_listen.

As I previously indicated his as an HA configuration, with 3 controllers with the current partial configuration on the /etc/haproxy/haproxy.cfg

listen nova-api-1
  balance  roundrobin
  option  httplog
  server  controller-01   check
  server  controller-02   check
  server  controller-03   check

ec2_listen_port is not defined in the /etc/nova/nova.conf, however 8773 is the default port for nova-api, would anything change by adding it to the nova.conf?


4) rest query response from Chrome:

<response> <errors> <error> Unauthorized <message>Signature not provided</message> </error> </errors> <requestid>req-0fef8417-77b0-4b0f-9553-da2c5957e80b</requestid> </response>

and from an instance using wget (this is the same instance I am running euca2tools)

ubuntu@instance02:~$ wget
--2013-10-28 15:59:47--
Connecting to connected.
HTTP request sent, awaiting response... 400 Bad Request
2013-10-28 15:59:48 ERROR 400: Bad Request.

ubuntu@instance02:~$ curl -skv -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" " ...
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted

answered 2013-10-29 04:14:16 -0600

kanaderohan gravatar image

Ok, the openstack confs look good.

Looking at the Openstack request id in your curl command output, it seems that your request is reaching 1 of the controllers.

<response><errors><error>500<message>The server has either erred or is incapable of performing the requested operation.</message></error></errors><requestid>req-de28fee4-2af4-428b-a202-4f8a2dbcf0f2</requestid></response>

Please do a grep command on all the openstack log files on each of your 3 controllers and post back the result

grep -r "req-de28fee4-2af4-428b-a202-4f8a2dbcf0f2" /path/to/your/openstack/logs

You can make another curl request and use that request id "req-xxx-xxx.." if you want to.

edit flag offensive delete link more


kanaderohan, I updated the ticket. Unfortunately as you can see no logs associated with the new curl request. Or logs are not generated, or this request gets lost somewhere (I believe first option is the right one)

LDC gravatar imageLDC ( 2013-10-29 04:40:04 -0600 )edit

I think the problem here is that, your logging config is not saving all the logs correctly. Are you using syslog? Normal files ? Can you make sure that your logging is configured correctly. Or just setup your Openstack to use Normal log files in /var/log/nova/. Then you can try to reproduce the same steps and using grep to find the logs.

kanaderohan gravatar imagekanaderohan ( 2013-10-29 04:53:37 -0600 )edit

Do you have any idea in which log file I should find the req-xxx-xxx among the nova*.log?

LDC gravatar imageLDC ( 2013-10-29 09:12:30 -0600 )edit

Could anybody provide some hint? I can see ERROR level logs for different services, however I am not able to find anything relevant to this issue. As I said in the ticket I can see the INFO log on the nova.ec2 log, however I dont have a clear idea what is the follow step after a ec2 API call? Providing an invalid username would return a HTTP 400 error, therefore I assume keystone is working as expected. Any suggestion would be very welcome. Thanks

LDC gravatar imageLDC ( 2013-10-31 03:18:05 -0600 )edit

answered 2013-10-28 08:16:45 -0600

kanaderohan gravatar image

1) Check if the (downloaded from horizon) provides the right credentials like EC2_AUTH, EC2_SECRET_KEY etc.

2) Check if you have the the correct /etc/nova/api-paste.ini (composite:ec2, composite:ec2cloud) 3) Check if /etc/nova/nova.conf has these flags correctly set -- enabled_apis=ec2 -- ec2_listen_port=8773 (check if this port is open/correctly bounded)

3) Also check the same api call using a rest query to http://your_endpoint:8773/services/Cloud

Report back the results so we can debug this issue further.

edit flag offensive delete link more


Kanaderohan, I updated my question with your requests. Thank you

LDC gravatar imageLDC ( 2013-10-28 10:48:53 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2013-10-28 00:47:24 -0600

Seen: 863 times

Last updated: Oct 29 '13