Ask Your Question
1

Failed to create instance, Nova api error timed out

asked 2014-06-24 20:26:04 -0600

alraouf gravatar image

updated 2014-06-24 20:29:34 -0600

Hi All,

I'm having a problem creating new instances, new install based on Icehouce, first instance to be created, all services looks ok and I can create networks/users/keys ..etc

Error during instance creation:

root@control:~# grep -ri error /var/log/*
/var/log/nova/nova-api.log:2014-06-24 18:01:42.505 7510 ERROR nova.api.openstack [req-adb059bb-4e75-455f-bdd5-80e8f1dbb6ca fb70a634a6c64daeb59f2cdc064177d6 5e9ffc4973fd4aedbe207d6bc833e470] Caught error: Timed out waiting for a reply to message ID 895650015eb140f49b8b9501a2ffb477
/var/log/upstart/nova-api.log:2014-06-24 18:01:42.505 7510 ERROR nova.api.openstack [req-adb059bb-4e75-455f-bdd5-80e8f1dbb6ca fb70a634a6c64daeb59f2cdc064177d6 5e9ffc4973fd4aedbe207d6bc833e470] Caught error: Timed out waiting for a reply to message ID 895650015eb140f49b8b9501a2ffb477

Note:

Same error from GUI or command line.

Following Installation guide here http://docs.openstack.org/icehouse/install-guide/install/apt/content/index.html (http://docs.openstack.org/icehouse/in...)

First time installing Icehouse

Ubuntu 12.04.4 (latest update), kernel 3.11.0-23-generic Nova version: nova-* --> 1:2014.1-0ubuntu1~cloud0

command I used

#nova --debug boot --image cirros-0.3.2-x86_64 --flavor m1.tiny --nic net-id=a0fbbc84-eb1d-439a-b6b8-54828a415cf3 instanceX

Trace:

REQ: curl -i 'http://control.xxx.com:8774/v2/5e9ffc4973fd4aedbe207d6bc833e470/servers' -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: ... omitted_text .." -d '{"server": {"name": "instanceX", "imageRef": "51a5d778-e962-4831-88d5-b77c35d41ec1", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "a0fbbc84-eb1d-439a-b6b8-54828a415cf3"}]}}'

DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "POST /v2/5e9ffc4973fd4aedbe207d6bc833e470/servers HTTP/1.1" 500 128
RESP: [500] CaseInsensitiveDict({'date': 'Wed, 25 Jun 2014 01:14:00 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-e2bcf9ef-9f38-46a3-a51d-a1d2f1b445d4'})
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:777) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-e2bcf9ef-9f38-46a3-a51d-a1d2f1b445d4)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 433, in do_boot
    server = cs.servers.create(*boot_args, **boot_kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 871, in create
    **boot_kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 534, in _boot
    return_raw=return_raw, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
    raise exceptions.from_response(resp, body, url, method)
**ClientException: The server has either erred or is incapable of performing the ...
(more)
edit retag flag offensive close merge delete

Comments

dbaxps gravatar imagedbaxps ( 2014-06-24 21:33:24 -0600 )edit

Thanks for the link, my error is different, my RPC is working fine, the time out I have has to do with message ID or rabbit queue somewhow

... Caught error: Timed out waiting for a reply to message ID ....

alraouf gravatar imagealraouf ( 2014-06-25 12:35:51 -0600 )edit

no errors in compute.log and conductor.log, libvirtd.log is empty.

I'm really stuck here, anyway to debug this more? debug is set to True already but no much help

alraouf gravatar imagealraouf ( 2014-06-25 14:03:47 -0600 )edit

There are lots of questions and answers with that specific error. Search and edit yours to highlight why your problem is different.

smaffulli gravatar imagesmaffulli ( 2014-06-25 15:08:44 -0600 )edit

@smaffulli, I did search and no luck so far, as I mentioned this is not the RPC timing out. Here is the error:

/var/log/nova/nova-api.log:2014-06-25 10:21:17.181 4620 ERROR nova.api.openstack [req-0be1...] Caught error: Timed out waiting for a reply to message ID da9.....

alraouf gravatar imagealraouf ( 2014-06-25 17:17:08 -0600 )edit

3 answers

Sort by ยป oldest newest most voted
1

answered 2014-06-25 17:53:34 -0600

smaffulli gravatar image
edit flag offensive delete link more

Comments

I went through all similar issues but nothing that applied to my issue, also reviewed all nova.conf, neutron.conf and look good as far as I can tell, changed to debug=true for nova and neutron, reinstalled nova, other nova command works such as nova list, nova image-list, neutron commands works

alraouf gravatar imagealraouf ( 2014-06-26 15:04:52 -0600 )edit

If I remove "--nic net-id=" the command works but the instance is stuck at "Build" state for ever.

Is there any way to find out what nova is waiting/timing out for? debug is not showing any helpful logs here, also other logs such as libvirtd and compute not showing any errors :(

alraouf gravatar imagealraouf ( 2014-06-26 15:07:30 -0600 )edit
0

answered 2014-06-25 13:31:00 -0600

mpetason gravatar image

Check the compute.log and the conductor log. See if anything is relevant with errors. RPC timeouts or anything related to that. Maybe "malformed message" or errors outside of Nova such as libvirt errors.

edit flag offensive delete link more

Comments

How can I check if it is a "malformed message"?

Looks like I will be reinstalling but this time Ubuntu 14.04 will be used instead of Ubuntu 12.

Thank you for your help!

alraouf gravatar imagealraouf ( 2014-06-27 13:16:54 -0600 )edit

Something that nova-compute chokes on. This was more relevant with older versions of rabbit/kombu libraries. In compute you'll see a traceback coming in after a message has been read. Traceback -- tons of info - last time has the actual error.

mpetason gravatar imagempetason ( 2014-06-27 13:24:18 -0600 )edit

Did a reinstall and the issue is not showing, the only difference here is Ubuntu 14.04, all looks good now

alraouf gravatar imagealraouf ( 2014-06-30 15:20:02 -0600 )edit
0

answered 2014-06-30 15:21:48 -0600

alraouf gravatar image

After reinstall I'm not see the problem again, thanks everyone for your help!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

4 followers

Stats

Asked: 2014-06-24 20:26:04 -0600

Seen: 15,615 times

Last updated: Jun 30 '14