Ask Your Question
0

nova-compute won't start anymore

asked 2014-11-11 15:43:41 -0600

federic0 gravatar image

updated 2014-11-14 01:15:38 -0600

UPDATE:

starting from command line with python --debug behaves as usual, 

2014-11-14 07:54:47.287 1125 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:65

2014-11-14 07:54:47.415 1125 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver' 2014-11-14 07:54:47.421 1125 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2014-11-14 07:54:47.470 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connecting to AMQP server on controller:5672 2014-11-14 07:54:47.489 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connected to AMQP server on controller:5672 2014-11-14 07:54:47.492 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connecting to AMQP server on controller:5672 2014-11-14 07:54:47.504 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connected to AMQP server on controller:5672

this happens immediately, until i press CRTL+C and so i get some TRACE lines:

2014-11-14 07:58:01.814 1125 CRITICAL nova [req-5074168d-4800-4ac9-a96e-a30c1ef28146 None] KeyboardInterrupt 2014-11-14 07:58:01.814 1125 TRACE nova Traceback (most recent call last): 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/bin/nova-compute", line 10, in <module> 2014-11-14 07:58:01.814 1125 TRACE nova sys.exit(main()) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main 2014-11-14 07:58:01.814 1125 TRACE nova db_allowed=CONF.conductor.use_local) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 275, in create 2014-11-14 07:58:01.814 1125 TRACE nova db_allowed=db_allowed) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 157, in __init__ 2014-11-14 07:58:01.814 1125 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context()) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 313, in wait_until_ready 2014-11-14 07:58:01.814 1125 TRACE nova timeout=timeout) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/baserpc.py", line 62, in ping 2014-11-14 07:58:01.814 1125 TRACE nova return cctxt.call(context, 'ping', arg=arg_p) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2014-11-14 07:58:01.814 1125 TRACE nova retry=self.retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-11-14 07:58:01.814 1125 TRACE nova timeout=timeout, retry=retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2014-11-14 07:58:01.814 1125 TRACE nova retry=retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo ... (more)

edit retag flag offensive close merge delete
add a comment

3 answers

Sort by ยป oldest newest most voted
1

answered 2014-11-17 14:51:54 -0600

federic0 gravatar image

i can't believe i spent so much time on this, but flushing iptables rules made nova-compute starting. thanks Herry for pointing me over there. now i need to understand which rule exactly avoid (or broke) rabbitmq connectivity.

edit flag offensive delete link more

Comments

This shouldn't have happened.Do you have this line in your nova.conf in both compute and controller firewall_driver=nova.virt.firewall.NoopFirewallDriver also you have to set correct neutron firewall and it will applies the correct ip tables but flushing isn't the right thing to do

bishoy gravatar imagebishoy ( 2014-11-18 08:28:44 -0600 )edit

i didn't yet because if i follow the official installation guide the step where the firewall driver must be added comes later, configuration of compute/controller nodes is in chapter 5, neutron starts from chapter 6. so, no i don't have it yet.

federic0 gravatar imagefederic0 ( 2014-11-18 10:17:16 -0600 )edit

moreover, there's another modification i have to apply any time i add a DB in mysql, GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'NOVA_DBPASS'; is ok, but i need to add it also for 'nova@controller' otherwise i get no access to DB; and this is valid for keystone, glance etc

federic0 gravatar imagefederic0 ( 2014-11-18 10:19:20 -0600 )edit

anyway, adding the firewll_driver does not make nova-compute starting, becasue amqp protocol is always blocked by iptables, syn comes from compute node, but no ack comes back from controller on port 5672. where i can troubleshoot those iptables rules? which is the component that sets iptables rules?

federic0 gravatar imagefederic0 ( 2014-11-18 14:21:17 -0600 )edit
add a comment
0

answered 2014-11-11 19:51:31 -0600

Eric Xie gravatar image

You can try to start nova-compute from daemon to foreground. Follow this: 1. systemctl stop openstack-nova-compute.service 2. python /usr/bin/nova-compute --debug Then check if error exist. Wish help~

edit flag offensive delete link more

Comments

OK thanks, i will try as soon as possible and let you know.

federic0 gravatar imagefederic0 ( 2014-11-12 05:16:48 -0600 )edit

I also encounter this error. When I execute the command " python /usr/bin/nova-compute --debug " , the console hang up always. Anyone has any idea?

herry gravatar imageherry ( 2014-11-14 02:30:16 -0600 )edit

me too, check the logs under /var/log/nova and let me know if you see rabbitmq intermittent connectivity

federic0 gravatar imagefederic0 ( 2014-11-15 01:29:04 -0600 )edit

the nova-compute.log always output the error : AMQP server on controller:5672 is unreachable. But I can ping the domain controller, telnet controller 5672 says "no route to host". I can telnet controller 5672 on controller node self. controller and compute node has multi nics and use suse centos7

herry gravatar imageherry ( 2014-11-15 20:36:23 -0600 )edit

While I use the iptables -F to clear the iptable at last,everything is ok. the openstack-nova-compute.service can be started

herry gravatar imageherry ( 2014-11-16 22:23:15 -0600 )edit
see more comments
0

answered 2014-11-11 17:03:57 -0600

dgalvao gravatar image

updated 2014-11-12 14:58:44 -0600

Update: maybe this helps you.

The issue I was seeing was caused by the /var filesystem dropping below 1GB and rabbitmq blocking publishers. From the rabbitmq logs:

=INFO REPORT==== 12-Nov-2014::15:14:32 ===
Disk free space insufficient. Free bytes:696029184 Limit:1000000000

=WARNING REPORT==== 12-Nov-2014::15:14:32 ===
disk resource limit alarm set on node 'rabbit@cha-ops-xen-v04'.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

I'm having a similar issue. I'm following that guide too and got the controller node running on a VM with keystone, glance and nova (minus compute) and all worked fine. However, on the compute node (running on a real server) I can't get the nova-compute service to start at all. It just hangs there with few helpful log messages even with the debug option on.

If you just run it as a service you will notice that it will timeout at about 90 seconds and try to start again and will just stay in that loop. For troubleshooting, I'm trying to start it manually: /usr/bin/nova-compute --debug --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log

You say "it won't start anymore". Did you ever get it to start successfully?

I've asked the same question a couple of hours ago, but it's waiting moderation. Hopefully we will have an answer soon.

edit flag offensive delete link more

Comments

it started once! the installation guide let's you verify the part where you were working on, and at the end of chapter 5 "verify operation" was successful. then i moved ahead on neutron where i've got another issue (neutron agent-list on the controller node gives nothing) and i discovered this issue

federic0 gravatar imagefederic0 ( 2014-11-12 05:16:09 -0600 )edit

unfortunatley i got no disk space issues.

federic0 gravatar imagefederic0 ( 2014-11-14 01:16:38 -0600 )edit
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

subscribe to rss feed

Stats

Asked: 2014-11-11 15:43:41 -0600

Seen: 1,855 times

Last updated: Nov 17 '14

Related questions

nova-compute rabbitmq fail to connect

Nova-Compute Ceph backend issues

Compute node services fail to connect to controller rabbitmq

ERROR oslo.messaging._drivers.impl_rabbit [req-b15a657e-8c7a-486d-9a80-7f6013724353 ] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. [closed]

ironic with keystone v3

Kolla-build failures on base_arch=aarch64

where the scheduling is done

Documentation for Server status ?

nova-network state XXX

How are actions on VMs handled?