Instance stuck on task_state powering-on when my router doesn't have internet access

asked 2018-09-25 19:47:44 -0500

Silvio Silva gravatar image

updated 2018-09-27 11:54:56 -0500

I've installed Openstack Queens using PackStack on top of a Intel NUC.

I've used a Cisco LinkSys E900 router to create a Public Network with 192.168.1.0/24 CIDR using those commands:

openstack network create \
--share \
--external \
--provider-network-type flat \
--provider-physical-network physnet1 \
public_network

openstack subnet create \
--dhcp \
--subnet-range 192.168.1.0/24 \
--allocation-pool start=192.168.1.20,end=192.168.1.90 \
--dns-nameserver 192.168.1.1 \
--network public_network \
public_subnet

I'm also created an internal network and subnet using this:

openstack network create \
--internal \
--provider-network-type vxlan \
devel_net

openstack subnet create \
--subnet-range 10.0.0.0/24 \
--allocation-pool start=10.0.0.20,end=10.0.0.90 \
--gateway 10.0.0.1 \
--dns-nameserver 192.168.1.1 \
--network devel_net \
devel_subnet

openstack router create \
--project devel \
devel_router1

openstack router add subnet \
devel_router1 \
devel_subnet

neutron router-gateway-set devel_router1 public_network

To create a Instance:

openstack_public_network_id=$(openstack network show public_network -c id -f value)

openstack_internal_network_id=$(openstack network show devel_net -c id -f value)

openstack_internal_subnet_id=$(openstack subnet show devel_subnet -c id -f value)

openstack_project_id=devel

openstack port create \
--disable-port-security \
--fixed-ip subnet=$openstack_internal_subnet_id,ip-address=10.0.0.41 \
--network $openstack_internal_network_id \
port-node-1

openstack floating ip create \
--project $openstack_project_id \
--port port-node-1 \
--floating-ip-address 192.168.1.41 \
$openstack_public_network_id

openstack server create \
--image centos7 \
--flavor m1.large \
--key-name director \
--port port-node1 \
node-1

Everything goes well and both creation and start process for multiples instances worked well with internet access but, my desire is to be able to use this environment even when I don't have internet access so I did a test disconnecting the cable from WAN port of my router and I'm not be able to start any instance anymore. The instance stay into poweroff state and task_state stays as powering-on until I connect the cable on router WAN port again. After I did it, everything comes to work fine again.

How can I troubleshoot this problem?

Thanks,

Silvio

Edit after Bernd tip (thanks Bernd):

I'm got those logs below, what requisition should I use to find accross Neutron logs?

Those logs were generated after command:

openstack server start node-1

Finding information using Instance ID (Internet Access On):

[root@openstack nova(cloud-admin)]# pwd
**/var/log/nova**
[root@openstack nova(cloud-admin)]# grep -R f5924fd1-c3c5-4d03-8da3-f9c77751bf68
nova-api.log:2018-09-27 13:12:15.892 3112 INFO nova.osapi_compute.wsgi.server [req-8342a1fa-30d5-4670-8cf4-0b42d60d0b27 53ca84b75a6f464e92043405443470d4 a1b9bf94403d4e6b9fdb90051e517841 - default default] 192.168.1.101 "GET /v2.1/a1b9bf94403d4e6b9fdb90051e517841/servers/f5924fd1-c3c5-4d03-8da3-f9c77751bf68 HTTP/1.1" status: 200 len: 2178 time: 0.2351341
nova-api.log:2018-09-27 13:12:16.026 3112 INFO nova.osapi_compute.wsgi.server [req-175ca5f0-1e91-4189-a60f-eaa845cc9c52 53ca84b75a6f464e92043405443470d4 a1b9bf94403d4e6b9fdb90051e517841 - default default] 192.168.1.101 "POST /v2.1/a1b9bf94403d4e6b9fdb90051e517841/servers/f5924fd1-c3c5-4d03-8da3-f9c77751bf68/action HTTP/1.1" status: 202 len: 403 time: 0.1310401
nova-api.log:2018-09-27 13:12:17.562 3112 INFO nova.api.openstack.compute.server_external_events [req-72b4bd53-3b49-46fe-94bb-068ef21b55fb b22cfc10d14e4bf596fb464c9b2ac701 9288b5f02add4a7faee3f21926df5ee4 - default default] Creating event network-vif-unplugged:09159689-cac0-4802-8eae-02a386fc8613 for instance f5924fd1-c3c5-4d03-8da3-f9c77751bf68 on openstack.example.com
nova-api.log:2018-09-27 13:12:18.432 3114 INFO nova.api.openstack.compute.server_external_events [req-ef48552e-7384-4829-9571-d951f0c86644 b22cfc10d14e4bf596fb464c9b2ac701 9288b5f02add4a7faee3f21926df5ee4 - default default] Creating event ...
(more)
edit retag flag offensive close merge delete

Comments

This is odd. To troubleshoot instance startup, you should inspect the logs. In this case perhaps nova-api, nova-compute, neutron-server and l3_agent logs.

You can search the Nova logs for the instance ID. In there, you find the Neutron API request ID for better analyzing Neutron logs.

Bernd Bausch gravatar imageBernd Bausch ( 2018-09-26 19:08:18 -0500 )edit

Hi Bernd, thanks for your reply.

Since I get some logs from before and after disconnect the cable from WAN port, I didn't know what information to find on Neutron logs.

Should I use UUID with initial "req-" like in "INFO nova.compute.manager [req-175ca5f0-1e91-4189-a60f-eaa845cc9c52"?

Thanks!

Silvio Silva gravatar imageSilvio Silva ( 2018-09-27 11:54:03 -0500 )edit

The “req-“ numbers identify API requests. I hoped to see Neutron API requests in Nova logs, but I think you need DEBUG logging for that.

Nova-comput says “rebooted successfully”. I wonder if the problem is on the instance rather than Nova or Neutron.

Bernd Bausch gravatar imageBernd Bausch ( 2018-09-27 16:53:16 -0500 )edit