Ask Your Question
0

euca2ools returns HTTP/1.1 500 Internal Server Error

asked 2013-10-28 00:47:24 -0500

LDC gravatar image

updated 2014-01-22 15:12:56 -0500

Evgeny gravatar image

Hello,

I have an Openstack deployment in HA, with 2 compute nodes and 3 controllers. Everything is working fine, however as soon as I try using EC2 Compatibility APIs though euca2ools commands I am always getting HTTP 500 error. nova*.log do not show anything that could help me pointing out where the problem could be. Native openstack calls works properly. I haven't find much documentation on how troubleshooting this issue and I would appreciate if anybody can provide some hints, ideas.

Here few information gathered from the controllers

1) ec2 credential in keystone ec-credentials-list --user-id match with the EC2 Access and Secret keys. EC2_AUTH is not set as not not defined in my ec2rc.sh. I did not have to create keystone credentials as many tutorial describe as it seems they are created when downloading the .zip file

2) api-paste.ini is the same in the 3 controllers, I paste here the config for ec2 and ec2cloud

#######
# EC2 #
#######

[composite:ec2]
use = egg:Paste#urlmap
/services/Cloud: ec2cloud

[composite:ec2cloud]
use = call:nova.api.auth:pipeline_factory
noauth = ec2faultwrap logrequest ec2noauth cloudrequest validator ec2executor
keystone = ec2faultwrap logrequest ec2keystoneauth cloudrequest validator ec2executor

[filter:ec2faultwrap]
paste.filter_factory = nova.api.ec2:FaultWrapper.factory

[filter:logrequest]
paste.filter_factory = nova.api.ec2:RequestLogging.factory

[filter:ec2lockout]
paste.filter_factory = nova.api.ec2:Lockout.factory

[filter:ec2keystoneauth]
paste.filter_factory = nova.api.ec2:EC2KeystoneAuth.factory

[filter:ec2noauth]
paste.filter_factory = nova.api.ec2:NoAuth.factory

[filter:cloudrequest]
controller = nova.api.ec2.cloud.CloudController
paste.filter_factory = nova.api.ec2:Requestify.factory

[filter:authorizer]
paste.filter_factory = nova.api.ec2:Authorizer.factory

[filter:validator]
paste.filter_factory = nova.api.ec2:Validator.factory

[app:ec2executor]
paste.app_factory = nova.api.ec2:Executor.factory

3) In /etc/nova/nova.conf enabled_apis include ec2, however in my nova.conf ec2_listen_port=8873 is not present but only ec2_listen.

As I previously indicated his as an HA configuration, with 3 controllers with the current partial configuration on the /etc/haproxy/haproxy.cfg

listen nova-api-1
  bind 172.16.0.254:8773
  bind 192.168.1.254:8773
  balance  roundrobin
  option  httplog
  server  controller-01 192.168.1.231:8773   check
  server  controller-02 192.168.1.232:8773   check
  server  controller-03 192.168.1.233:8773   check

ec2_listen_port is not defined in the /etc/nova/nova.conf, however 8773 is the default port for nova-api, would anything change by adding it to the nova.conf?

enabled_apis=ec2,osapi_compute,

4) rest query response from Chrome:

<response> <errors> <error> Unauthorized <message>Signature not provided</message> </error> </errors> <requestid>req-0fef8417-77b0-4b0f-9553-da2c5957e80b</requestid> </response>

and from an instance using wget (this is the same instance I am running euca2tools)

ubuntu@instance02:~$ wget http://172.16.0.254:8773/services/Cloud/
--2013-10-28 15:59:47--  http://172.16.0.254:8773/services/Cloud/
Connecting to 172.16.0.254:8773... connected.
HTTP request sent, awaiting response... 400 Bad Request
2013-10-28 15:59:48 ERROR 400: Bad Request.

ubuntu@instance02:~$
ubuntu@instance02:~$ curl -skv -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" "http://172.16.0.254:8773/services/Cloud/?AWSAccessKeyId ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2013-10-29 04:14:16 -0500

kanaderohan gravatar image

Ok, the openstack confs look good.

Looking at the Openstack request id in your curl command output, it seems that your request is reaching 1 of the controllers.

<response><errors><error>500<message>The server has either erred or is incapable of performing the requested operation.</message></error></errors><requestid>req-de28fee4-2af4-428b-a202-4f8a2dbcf0f2</requestid></response>

Please do a grep command on all the openstack log files on each of your 3 controllers and post back the result

grep -r "req-de28fee4-2af4-428b-a202-4f8a2dbcf0f2" /path/to/your/openstack/logs

You can make another curl request and use that request id "req-xxx-xxx.." if you want to.

edit flag offensive delete link more

Comments

kanaderohan, I updated the ticket. Unfortunately as you can see no logs associated with the new curl request. Or logs are not generated, or this request gets lost somewhere (I believe first option is the right one)

LDC gravatar imageLDC ( 2013-10-29 04:40:04 -0500 )edit

I think the problem here is that, your logging config is not saving all the logs correctly. Are you using syslog? Normal files ? Can you make sure that your logging is configured correctly. Or just setup your Openstack to use Normal log files in /var/log/nova/. Then you can try to reproduce the same steps and using grep to find the logs.

kanaderohan gravatar imagekanaderohan ( 2013-10-29 04:53:37 -0500 )edit

Do you have any idea in which log file I should find the req-xxx-xxx among the nova*.log?

LDC gravatar imageLDC ( 2013-10-29 09:12:30 -0500 )edit

Could anybody provide some hint? I can see ERROR level logs for different services, however I am not able to find anything relevant to this issue. As I said in the ticket I can see the INFO log on the nova.ec2 log, however I dont have a clear idea what is the follow step after a ec2 API call? Providing an invalid username would return a HTTP 400 error, therefore I assume keystone is working as expected. Any suggestion would be very welcome. Thanks

LDC gravatar imageLDC ( 2013-10-31 03:18:05 -0500 )edit
0

answered 2013-10-28 08:16:45 -0500

kanaderohan gravatar image

1) Check if the ec2rc.sh (downloaded from horizon) provides the right credentials like EC2_AUTH, EC2_SECRET_KEY etc.

2) Check if you have the the correct /etc/nova/api-paste.ini (composite:ec2, composite:ec2cloud) 3) Check if /etc/nova/nova.conf has these flags correctly set -- enabled_apis=ec2 -- ec2_listen_port=8773 (check if this port is open/correctly bounded)

3) Also check the same api call using a rest query to http://your_endpoint:8773/services/Cloud

Report back the results so we can debug this issue further.

edit flag offensive delete link more

Comments

Kanaderohan, I updated my question with your requests. Thank you

LDC gravatar imageLDC ( 2013-10-28 10:48:53 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-10-28 00:47:24 -0500

Seen: 819 times

Last updated: Oct 29 '13