Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Nova instance 500 error

HI all-

So Ive gotten up to the point where I am to launch an instance (3 node setup - controller, network, compute) and I am getting this error below. Ive been searching (mailing list, ask openstack, google) for the past two days for any explanations or ways to fix and Im at wits end - especially since this is the most generic error ever. Would anyone have ideas where to look for this? Thank you very much!

REQ: curl -i 'http://controller:8774/v2/0e272c64703d408fab77665fabc6e7e6/servers' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}0a6884f08c3d40df8fe484658d7a6bfb9702d2c0" -d '{"server": {"name": "demo1", "imageRef": "c7590319-8719-4612-b959-8bffd405da3a", "key_name": "demo-key", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "4b27737a-a063-4b6e-b21a-8cffab8e3dc5"}], "security_groups": [{"name": "default"}]}}' INFO (connectionpool:187) Starting new HTTP connection (1): controller

DEBUG (connectionpool:357) "POST /v2/0e272c64703d408fab77665fabc6e7e6/servers HTTP/1.1" 500 128

RESP: [500] {'date': 'Wed, 19 Nov 2014 22:29:41 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'}

RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:803) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 470, in do_boot server = cs.servers.create(boot_args, *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 902, in create *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 554, in _boot return_raw=return_raw, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post return self._cs_request(url, 'POST', *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request resp, body = self._time_request(url, method, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 433, in request raise exceptions.from_response(resp, body, url, method)

ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Nova instance creation 500 error

HI all-

So Ive gotten up to the point where I am to launch an instance (3 node setup - controller, network, compute) and I am getting this error below. Ive been searching (mailing list, ask openstack, google) for the past two days for any explanations or ways to fix and Im at wits end - especially since this is the most generic error ever. Would anyone have ideas where to look for this? Thank you very much!

REQ: curl -i 'http://controller:8774/v2/0e272c64703d408fab77665fabc6e7e6/servers' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}0a6884f08c3d40df8fe484658d7a6bfb9702d2c0" -d '{"server": {"name": "demo1", "imageRef": "c7590319-8719-4612-b959-8bffd405da3a", "key_name": "demo-key", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "4b27737a-a063-4b6e-b21a-8cffab8e3dc5"}], "security_groups": [{"name": "default"}]}}' INFO (connectionpool:187) Starting new HTTP connection (1): controller

DEBUG (connectionpool:357) "POST /v2/0e272c64703d408fab77665fabc6e7e6/servers HTTP/1.1" 500 128

RESP: [500] {'date': 'Wed, 19 Nov 2014 22:29:41 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'}

RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:803) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 470, in do_boot server = cs.servers.create(boot_args, *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 902, in create *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 554, in _boot return_raw=return_raw, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post return self._cs_request(url, 'POST', *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request resp, body = self._time_request(url, method, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 433, in request raise exceptions.from_response(resp, body, url, method)

ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Nova instance creation 500 error

HI all-

So Ive gotten up to the point where I am to launch an instance (3 node setup - controller, network, compute) and I am getting this error below. Ive been searching (mailing list, ask openstack, google) for the past two days for any explanations or ways to fix and Im at wits end - especially since this is the most generic error ever. Would anyone have ideas where to look for this? Thank you very much!

nova --debug boot --flavor m1.tiny --image cirros-0.3.3-x86_64 --nic net-id=4b27737a-a063-4b6e-b21a-8cffab8e3dc5 --security-group default --key-name demo-key demo1

REQ: curl -i 'http://controller:8774/v2/0e272c64703d408fab77665fabc6e7e6/servers' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}0a6884f08c3d40df8fe484658d7a6bfb9702d2c0" -d '{"server": {"name": "demo1", "imageRef": "c7590319-8719-4612-b959-8bffd405da3a", "key_name": "demo-key", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "4b27737a-a063-4b6e-b21a-8cffab8e3dc5"}], "security_groups": [{"name": "default"}]}}' INFO (connectionpool:187) Starting new HTTP connection (1): controller

DEBUG (connectionpool:357) "POST /v2/0e272c64703d408fab77665fabc6e7e6/servers HTTP/1.1" 500 128

RESP: [500] {'date': 'Wed, 19 Nov 2014 22:29:41 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'}

RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:803) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 470, in do_boot server = cs.servers.create(boot_args, *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 902, in create *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 554, in _boot return_raw=return_raw, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post return self._cs_request(url, 'POST', *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request resp, body = self._time_request(url, method, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 433, in request raise exceptions.from_response(resp, body, url, method)

ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

This is the error that shows up in nova-api.log:

2014-11-20 17:58:55.935 9225 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 47e5a92644d94d1d898fd0a4c80604ac

Nova instance creation 500 error

HI all-

So Ive gotten up to the point where I am to launch an instance (3 node setup - controller, network, compute) and I am getting this error below. Ive been searching (mailing list, ask openstack, google) for the past two days for any explanations or ways to fix and Im at wits end - especially since this is the most generic error ever. Would anyone have ideas where to look for this? Thank you very much!

nova --debug boot --flavor m1.tiny --image cirros-0.3.3-x86_64 --nic net-id=4b27737a-a063-4b6e-b21a-8cffab8e3dc5 --security-group default --key-name demo-key demo1

REQ: curl -i 'http://controller:8774/v2/0e272c64703d408fab77665fabc6e7e6/servers' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}0a6884f08c3d40df8fe484658d7a6bfb9702d2c0" -d '{"server": {"name": "demo1", "imageRef": "c7590319-8719-4612-b959-8bffd405da3a", "key_name": "demo-key", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "4b27737a-a063-4b6e-b21a-8cffab8e3dc5"}], "security_groups": [{"name": "default"}]}}' INFO (connectionpool:187) Starting new HTTP connection (1): controller

DEBUG (connectionpool:357) "POST /v2/0e272c64703d408fab77665fabc6e7e6/servers HTTP/1.1" 500 128

RESP: [500] {'date': 'Wed, 19 Nov 2014 22:29:41 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'}

RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

DEBUG (shell:803) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 470, in do_boot server = cs.servers.create(boot_args, *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 902, in create *boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 554, in _boot return_raw=return_raw, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post return self._cs_request(url, 'POST', *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request resp, body = self._time_request(url, method, *kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 433, in request raise exceptions.from_response(resp, body, url, method)

ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

This is the error that shows up in nova-api.log:

2014-11-20 17:58:55.935 9225 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 47e5a92644d94d1d898fd0a4c80604ac

**UPDATE: So I notice this in the nova-compute log on compute node:

2014-11-19 12:15:08.879 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: Socket closed

2014-11-19 12:15:08.879 1522 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed

2014-11-19 12:16:28.800 1522 ERROR nova.servicegroup.drivers.db [-] model server went away

2014-11-19 12:16:28.813 1522 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!

2014-11-19 14:32:51.826 1516 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)

2014-11-19 12:15:02.860 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.

2014-11-19 12:15:05.216 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.

But I can telnet from compute node to controller node on port 5672 just fine:

[root@compute1 nova]# telnet controller 5672

Trying 10.7.5.11...

Connected to controller.

Escape character is '^]'.

click to hide/show revision 5
No.5 Revision

Nova instance creation 500 error

HI all-

So Ive gotten up to the point where I am to launch an instance (3 node setup - controller, network, compute) and I am getting this error below. Ive been searching (mailing list, ask openstack, google) for the past two days for any explanations or ways to fix and Im at wits end - especially since this is the most generic error ever. Would anyone have ideas where to look for this? Thank you very much!

# nova --debug boot --flavor m1.tiny --image cirros-0.3.3-x86_64 --nic net-id=4b27737a-a063-4b6e-b21a-8cffab8e3dc5 --security-group default --key-name demo-key demo1

demo1 REQ: curl -i 'http://controller:8774/v2/0e272c64703d408fab77665fabc6e7e6/servers' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}0a6884f08c3d40df8fe484658d7a6bfb9702d2c0" -d '{"server": {"name": "demo1", "imageRef": "c7590319-8719-4612-b959-8bffd405da3a", "key_name": "demo-key", "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "4b27737a-a063-4b6e-b21a-8cffab8e3dc5"}], "security_groups": [{"name": "default"}]}}' INFO (connectionpool:187) Starting new HTTP connection (1): controller

controller DEBUG (connectionpool:357) "POST /v2/0e272c64703d408fab77665fabc6e7e6/servers HTTP/1.1" 500 128

128 RESP: [500] {'date': 'Wed, 19 Nov 2014 22:29:41 GMT', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'}

'req-07dfa570-0364-4ffd-ade7-ed84ed86abc7'} RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

500}} DEBUG (shell:803) The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) Traceback (most recent call last):

last): File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 800, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 730, in main args.func(self.cs, args) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/shell.py", line 470, in do_boot server = cs.servers.create(boot_args, *boot_kwargs) cs.servers.create(*boot_args, **boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 902, in create *boot_kwargs) **boot_kwargs) File "/usr/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 554, in _boot return_raw=return_raw, *kwargs) **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 100, in _create _resp, body = self.api.client.post(url, body=body) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 490, in post return self._cs_request(url, 'POST', *kwargs) **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 465, in _cs_request resp, body = self._time_request(url, method, *kwargs) **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 439, in _time_request resp, body = self.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 433, in request raise exceptions.from_response(resp, body, url, method)

method) ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7) ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

req-07dfa570-0364-4ffd-ade7-ed84ed86abc7)

This is the error that shows up in nova-api.log:

2014-11-20 17:58:55.935 9225 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 47e5a92644d94d1d898fd0a4c80604ac

47e5a92644d94d1d898fd0a4c80604ac

**UPDATE: So I notice this in the nova-compute log on compute node:

2014-11-19 12:15:08.879 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: Socket closed

closed 2014-11-19 12:15:08.879 1522 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed

closed 2014-11-19 12:16:28.800 1522 ERROR nova.servicegroup.drivers.db [-] model server went away

away 2014-11-19 12:16:28.813 1522 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!

connection! 2014-11-19 14:32:51.826 1516 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)

attempt(s) 2014-11-19 12:15:02.860 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.

seconds. 2014-11-19 12:15:05.216 1522 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.

seconds.

But I can telnet from compute node to controller node on port 5672 just fine:

[root@compute1 nova]# telnet controller 5672

5672 Trying 10.7.5.11...

10.7.5.11... Connected to controller.

controller. Escape character is '^]'.

'^]'.