Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

I am attempting in installing openstack on three servers:

  1. Controller
  2. Compute
  3. Network

I am making an effort in doing so following the instructions at http://docs.openstack.org/havana/install-guide/install/apt/content/index.html. I am using Ubuntu 13.10 on all three servers.

I followed the instructions up to installing the dashboard (http://docs.openstack.org/havana/install-guide/install/apt/content/install_dashboard.html) and am able to get to the dashboard. I used the neutron configuration with openvswitch network instructions as we are interested in implementing that and possibly openflow at some time.

I then set up the memcached as suggested. This part confused me a bit, but if I am reading it correctly there are three options for this. Local, memcached and Redis. I am assuming I need to pick one, so I picked memcached. (Please advise if that is incorrect).

I am now trying to install an instance in the dashboard interface and am getting the following error:

Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

I googled this a bit and found a bunch of references to authentication issues. I re-verified username and passwords with what I put into keystone and am still seeing this issue. I looked at the nova-api.log file and I am getting the following:

2014-03-17 15:50:35.585 2384 ERROR nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] Caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>" 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Traceback (most recent call last): 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return req.get_response(self.application) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack application, catch_exc_info=False) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return resp(environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return self.app(env, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return resp(environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return resp(environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack response = self.app(environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return resp(environ, start_response) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack resp = self.call_func(req, args, *self.kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return self.func(req, args, *kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 938, in __call__ 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack content_type, body, accept) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 997, in _process_stack 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return method(req=request, *action_args) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 920, in create 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack legacy_bdm=legacy_bdm) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack rv = f(args, *kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1222, in create 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack legacy_bdm=legacy_bdm) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 868, in _create_instance 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack block_device_mapping, auto_disk_config, reservation_id) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 682, in _validate_and_build_base_options 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack self._check_requested_networks(context, requested_networks) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 436, in _check_requested_networks 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack self.network_api.validate_networks(context, requested_networks) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 94, in wrapped 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return func(self, context, *args, *kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 422, in validate_networks 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack requested_networks) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 228, in validate_networks 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return self.client.call(ctxt, 'validate_networks', networks=networks) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return self._invoke(self.proxy.call, ctxt, method, *kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack return cast_or_call(ctxt, msg, *self.kwargs) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 130, in call 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack exc.info, real_topic, msg.get('method')) 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>" 2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack 2014-03-17 15:50:35.588 2384 INFO nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] http://controller:8774/v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers returned with HTTP 500 2014-03-17 15:50:35.590 2384 INFO nova.osapi_compute.wsgi.server [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] 127.0.0.1 "POST /v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers HTTP/1.1" status: 500 len: 335 time: 60.1182590

At this point, I am at a bit of a loss on where to go from here. Can someone advise me what I am doing wrong? I am sure it is something I did incorrectly or overlooked.

Thanks in advance

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

I am attempting in installing openstack on three servers:

  1. Controller
  2. Compute
  3. Network

I am making an effort in doing so following the instructions at http://docs.openstack.org/havana/install-guide/install/apt/content/index.html. I am using Ubuntu 13.10 on all three servers.

I followed the instructions up to installing the dashboard (http://docs.openstack.org/havana/install-guide/install/apt/content/install_dashboard.html) and am able to get to the dashboard. I used the neutron configuration with openvswitch network instructions as we are interested in implementing that and possibly openflow at some time.

I then set up the memcached as suggested. This part confused me a bit, but if I am reading it correctly there are three options for this. Local, memcached and Redis. I am assuming I need to pick one, so I picked memcached. (Please advise if that is incorrect).

I am now trying to install an instance in the dashboard interface and am getting the following error:

 Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

500)

I googled this a bit and found a bunch of references to authentication issues. I re-verified username and passwords with what I put into keystone and am still seeing this issue. I looked at the nova-api.log file and I am getting the following:

2014-03-17 15:50:35.585 2384 ERROR nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] Caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return req.get_response(self.application)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.app(env, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.func(req, args, *kwargs)
*args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 938, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     content_type, body, accept)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 997, in _process_stack
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return method(req=request, *action_args)
**action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 920, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     rv = f(args, *kwargs)
f(*args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1222, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 868, in _create_instance
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     block_device_mapping, auto_disk_config, reservation_id)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 682, in _validate_and_build_base_options
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self._check_requested_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 436, in _check_requested_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self.network_api.validate_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 94, in wrapped
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return func(self, context, *args, *kwargs)
**kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 422, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 228, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.client.call(ctxt, 'validate_networks', networks=networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self._invoke(self.proxy.call, ctxt, method, *kwargs)
**kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return cast_or_call(ctxt, msg, *self.kwargs)
**self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     exc.info, real_topic, msg.get('method'))
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack 
2014-03-17 15:50:35.588 2384 INFO nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] http://controller:8774/v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers returned with HTTP 500
2014-03-17 15:50:35.590 2384 INFO nova.osapi_compute.wsgi.server [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] 127.0.0.1 "POST /v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers HTTP/1.1" status: 500 len: 335 time: 60.1182590

60.1182590

At this point, I am at a bit of a loss on where to go from here. Can someone advise me what I am doing wrong? I am sure it is something I did incorrectly or overlooked.

Thanks in advance

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

I am attempting in installing openstack on three servers:

  1. Controller
  2. Compute
  3. Network

I am making an effort in doing so following followed the instructions at http://docs.openstack.org/havana/install-guide/install/apt/content/index.html. I am using Ubuntu 13.10 on all three servers.

I followed the instructions up to installing the dashboard (http://docs.openstack.org/havana/install-guide/install/apt/content/install_dashboard.html) and am able to get to the dashboard. I used the neutron configuration with openvswitch network instructions as we are interested in implementing that and possibly openflow at some time.

I then set up the memcached as suggested. This part confused me a bit, but if I am reading it correctly there are three options for this. Local, memcached and Redis. I am assuming I need to pick one, so I picked memcached. (Please advise if that is incorrect).

I am now trying to install an instance in the dashboard interface and am getting the following error:

  Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

I googled this a bit and found a bunch of references to authentication issues. I re-verified username and passwords with what I put into keystone and am still seeing this issue. I looked at the nova-api.log file and I am getting the following:

2014-03-17 15:50:35.585 2384 ERROR nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] Caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return req.get_response(self.application)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.app(env, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 938, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     content_type, body, accept)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 997, in _process_stack
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 920, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     rv = f(*args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1222, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 868, in _create_instance
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     block_device_mapping, auto_disk_config, reservation_id)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 682, in _validate_and_build_base_options
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self._check_requested_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 436, in _check_requested_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self.network_api.validate_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 94, in wrapped
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return func(self, context, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 422, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 228, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.client.call(ctxt, 'validate_networks', networks=networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return cast_or_call(ctxt, msg, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     exc.info, real_topic, msg.get('method'))
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack 
2014-03-17 15:50:35.588 2384 INFO nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] http://controller:8774/v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers returned with HTTP 500
2014-03-17 15:50:35.590 2384 INFO nova.osapi_compute.wsgi.server [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] 127.0.0.1 "POST /v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers HTTP/1.1" status: 500 len: 335 time: 60.1182590

At this point, I am at a bit of a loss on where to go from here. Can someone advise me what I am doing wrong? I am sure it is something I did incorrectly or overlooked.

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

I am attempting in installing openstack on three servers:

  1. Controller
  2. Compute
  3. Network

I followed the instructions at http://docs.openstack.org/havana/install-guide/install/apt/content/index.html. I am using Ubuntu 13.10 on all three servers.

I followed the instructions up to installing the dashboard (http://docs.openstack.org/havana/install-guide/install/apt/content/install_dashboard.html) and am able to get to the dashboard. I used the neutron configuration with openvswitch network instructions as we are interested in implementing that and possibly openflow at some time.

I then set up the memcached as suggested. This part confused me a bit, but if I am reading it correctly there are three options for this. Local, memcached and Redis. I am assuming I need to pick one, so I picked memcached. (Please advise if that is incorrect).

I am now trying to install an instance in the dashboard interface and am getting the following error:

  Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

I googled this a bit and found a bunch of references to authentication issues. I re-verified username and passwords with what I put into keystone and am still seeing this issue. I looked at the nova-api.log file and I am getting the following:

2014-03-17 15:50:35.585 2384 ERROR nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] Caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return req.get_response(self.application)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.app(env, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 938, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     content_type, body, accept)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 997, in _process_stack
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 920, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     rv = f(*args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1222, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 868, in _create_instance
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     block_device_mapping, auto_disk_config, reservation_id)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 682, in _validate_and_build_base_options
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self._check_requested_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 436, in _check_requested_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self.network_api.validate_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 94, in wrapped
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return func(self, context, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 422, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 228, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.client.call(ctxt, 'validate_networks', networks=networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return cast_or_call(ctxt, msg, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     exc.info, real_topic, msg.get('method'))
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack 
2014-03-17 15:50:35.588 2384 INFO nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] http://controller:8774/v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers returned with HTTP 500
2014-03-17 15:50:35.590 2384 INFO nova.osapi_compute.wsgi.server [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] 127.0.0.1 "POST /v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers HTTP/1.1" status: 500 len: 335 time: 60.1182590

At this point, I am at a bit of a loss on where to go from here. Can someone advise me what I am doing wrong? I am sure it is something I did incorrectly or overlooked.

Here is the contents of my nova.conf on the compute node:

   [DEFAULT]
    dhcpbridge_flagfile=/etc/nova/nova.conf
    dhcpbridge=/usr/bin/nova-dhcpbridge
    logdir=/var/log/nova
    state_path=/var/lib/nova
    lock_path=/var/lock/nova
    force_dhcp_release=True
    iscsi_helper=tgtadm
    libvirt_use_virtio_for_bridges=True
    connection_type=libvirt
    root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
    verbose=True
    ec2_private_dns_show_ip=True
    api_paste_config=/etc/nova/api-paste.ini
    volumes_path=/var/lib/nova/volumes
    enabled_apis=ec2,osapi_compute,metadata
    auth_strategy=keystone
    rpc_backend = nova.rpc.impl_kombu
    rabbit_host = controller
    rabbit_password = <removed>
    my_ip=10.0.2.86
    vnc_enabled=True
    vncserver_listen=0.0.0.0
    vncserver_proxyclient_address=10.0.2.86
    novncproxy_base_url=http://controller:6080/vnc_auto.html
    glance_host=controller
    network_api_class=nova.network.neutronv2.api.API
    neutron_url=http://controller:9696
    neutron_auth_strategy=keystone
    neutron_admin_tenant_name=service
    neutron_admin_username=neutron
    neutron_admin_password=<removed>
    neutron_admin_auth_url=http://controller:35357/v2.0
    linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
    firewall_driver=nova.virt.firewall.NoopFirewallDriver
    #security_group_api=neutron


    [database]
    The SQLAlchemy connection string used to connect to the database
    connection = mysql://nova:<removed>@controller/nova

Here is the nova.conf file on the controller node:

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
rpc_backend = nova.rpc.impl_kombu
rabbit_host = controller
rabbit_password = <removed>
my_ip=10.0.2.83
vncserver_listen=10.0.2.83
vncserver_proxyclient_address=10.0.2.83
auth_strategy=keystone
neutron_metadata_proxy_shared_secret = <removed>
service_neutron_metadata_proxy = true

####### added the following according to directions
[database]
# The SQLAlchemy connection string used to connect to the database
connection = mysql://nova:<removed>@controller/nova
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = <removed>

####neutron info
network_api_class=nova.network.neutronv2.api.API
neutron_url=http://controller:9696
neutron_auth_strategy=keystone
neutron_admin_tenant_name=service
neutron_admin_username=neutron
neutron_admin_password=<removed>
neutron_admin_auth_url=http://controller:35357/v2.0
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver=nova.virt.firewall.NoopFirewallDriver

#security_group_api=neutron

the neutron.conf file on the network node has the following, I will spare you looking at the whole file:

rabbit_host = controller
rabbit_userid = guest
rabbit_password = fmadmin

the neutron.conf file on the controller has the following:

fake_rabbit = False
rabbit_host = controller
rabbit_port = 5672
rabbit_password = fmadmin

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

I am attempting in installing openstack on three servers:

  1. Controller
  2. Compute
  3. Network

I followed the instructions at http://docs.openstack.org/havana/install-guide/install/apt/content/index.html. I am using Ubuntu 13.10 on all three servers.

I followed the instructions up to installing the dashboard (http://docs.openstack.org/havana/install-guide/install/apt/content/install_dashboard.html) and am able to get to the dashboard. I used the neutron configuration with openvswitch network instructions as we are interested in implementing that and possibly openflow at some time.

I then set up the memcached as suggested. This part confused me a bit, but if I am reading it correctly there are three options for this. Local, memcached and Redis. I am assuming I need to pick one, so I picked memcached. (Please advise if that is incorrect).

I am now trying to install an instance in the dashboard interface and am getting the following error:

  Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

I googled this a bit and found a bunch of references to authentication issues. I re-verified username and passwords with what I put into keystone and am still seeing this issue. I looked at the nova-api.log file and I am getting the following:

2014-03-17 15:50:35.585 2384 ERROR nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] Caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Traceback (most recent call last):
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return req.get_response(self.application)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 539, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.app(env, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return resp(environ, start_response)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 938, in __call__
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     content_type, body, accept)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 997, in _process_stack
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 920, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     rv = f(*args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1222, in create
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     legacy_bdm=legacy_bdm)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 868, in _create_instance
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     block_device_mapping, auto_disk_config, reservation_id)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 682, in _validate_and_build_base_options
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self._check_requested_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 436, in _check_requested_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     self.network_api.validate_networks(context, requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 94, in wrapped
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return func(self, context, *args, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 422, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     requested_networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 228, in validate_networks
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self.client.call(ctxt, 'validate_networks', networks=networks)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     return cast_or_call(ctxt, msg, **self.kwargs)
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 130, in call
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack     exc.info, real_topic, msg.get('method'))
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack Timeout: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"
2014-03-17 15:50:35.585 2384 TRACE nova.api.openstack 
2014-03-17 15:50:35.588 2384 INFO nova.api.openstack [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] http://controller:8774/v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers returned with HTTP 500
2014-03-17 15:50:35.590 2384 INFO nova.osapi_compute.wsgi.server [req-1bf10e55-3ff7-4c29-8165-7a6ad2a21c4c bb772ea1a561489fbeb7cb52050679c2 8f2b0c3bbdbb492299508c9ef9d909b3] 127.0.0.1 "POST /v2/8f2b0c3bbdbb492299508c9ef9d909b3/servers HTTP/1.1" status: 500 len: 335 time: 60.1182590

At this point, I am at a bit of a loss on where to go from here. Can someone advise me what I am doing wrong? I am sure it is something I did incorrectly or overlooked.

Here is the contents of my nova.conf on the compute node:

   [DEFAULT]
    dhcpbridge_flagfile=/etc/nova/nova.conf
    dhcpbridge=/usr/bin/nova-dhcpbridge
    logdir=/var/log/nova
    state_path=/var/lib/nova
    lock_path=/var/lock/nova
    force_dhcp_release=True
    iscsi_helper=tgtadm
    libvirt_use_virtio_for_bridges=True
    connection_type=libvirt
    root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
    verbose=True
    ec2_private_dns_show_ip=True
    api_paste_config=/etc/nova/api-paste.ini
    volumes_path=/var/lib/nova/volumes
    enabled_apis=ec2,osapi_compute,metadata
    auth_strategy=keystone
    rpc_backend = nova.rpc.impl_kombu
    rabbit_host = controller
    rabbit_password = <removed>
    my_ip=10.0.2.86
    vnc_enabled=True
    vncserver_listen=0.0.0.0
    vncserver_proxyclient_address=10.0.2.86
    novncproxy_base_url=http://controller:6080/vnc_auto.html
    glance_host=controller
    network_api_class=nova.network.neutronv2.api.API
    neutron_url=http://controller:9696
    neutron_auth_strategy=keystone
    neutron_admin_tenant_name=service
    neutron_admin_username=neutron
    neutron_admin_password=<removed>
    neutron_admin_auth_url=http://controller:35357/v2.0
    linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
    firewall_driver=nova.virt.firewall.NoopFirewallDriver
    #security_group_api=neutron


    [database]
    The SQLAlchemy connection string used to connect to the database
    connection = mysql://nova:<removed>@controller/nova

Here is the nova.conf file on the controller node:

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
rpc_backend = nova.rpc.impl_kombu
rabbit_host = controller
rabbit_password = <removed>
my_ip=10.0.2.83
vncserver_listen=10.0.2.83
vncserver_proxyclient_address=10.0.2.83
auth_strategy=keystone
neutron_metadata_proxy_shared_secret = <removed>
service_neutron_metadata_proxy = true

####### added the following according to directions
[database]
# The SQLAlchemy connection string used to connect to the database
connection = mysql://nova:<removed>@controller/nova
[keystone_authtoken]
auth_host = controller
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = <removed>

####neutron info
network_api_class=nova.network.neutronv2.api.API
neutron_url=http://controller:9696
neutron_auth_strategy=keystone
neutron_admin_tenant_name=service
neutron_admin_username=neutron
neutron_admin_password=<removed>
neutron_admin_auth_url=http://controller:35357/v2.0
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver=nova.virt.firewall.NoopFirewallDriver

#security_group_api=neutron

the neutron.conf file on the network node has the following, I will spare you looking at the whole file:

rabbit_host = controller
rabbit_userid = guest
rabbit_password = fmadmin
<removed>

the neutron.conf file on the controller has the following:

fake_rabbit = False
rabbit_host = controller
rabbit_port = 5672
rabbit_password = fmadmin
<removed>