nova-compute won't start anymore

asked 2014-11-11 15:43:41 -0500

federic0
36 ●3 ●4 ●7

updated 2014-11-14 01:15:38 -0500

UPDATE:

starting from command line with python --debug behaves as usual,

2014-11-14 07:54:47.287 1125 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:65

2014-11-14 07:54:47.415 1125 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver' 2014-11-14 07:54:47.421 1125 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2014-11-14 07:54:47.470 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connecting to AMQP server on controller:5672 2014-11-14 07:54:47.489 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connected to AMQP server on controller:5672 2014-11-14 07:54:47.492 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connecting to AMQP server on controller:5672 2014-11-14 07:54:47.504 1125 INFO oslo.messaging._drivers.impl_rabbit [req-5074168d-4800-4ac9-a96e-a30c1ef28146 ] Connected to AMQP server on controller:5672

this happens immediately, until i press CRTL+C and so i get some TRACE lines:

2014-11-14 07:58:01.814 1125 CRITICAL nova [req-5074168d-4800-4ac9-a96e-a30c1ef28146 None] KeyboardInterrupt 2014-11-14 07:58:01.814 1125 TRACE nova Traceback (most recent call last): 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/bin/nova-compute", line 10, in <module> 2014-11-14 07:58:01.814 1125 TRACE nova sys.exit(main()) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/cmd/compute.py", line 72, in main 2014-11-14 07:58:01.814 1125 TRACE nova db_allowed=CONF.conductor.use_local) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 275, in create 2014-11-14 07:58:01.814 1125 TRACE nova db_allowed=db_allowed) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 157, in __init__ 2014-11-14 07:58:01.814 1125 TRACE nova self.conductor_api.wait_until_ready(context.get_admin_context()) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 313, in wait_until_ready 2014-11-14 07:58:01.814 1125 TRACE nova timeout=timeout) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/nova/baserpc.py", line 62, in ping 2014-11-14 07:58:01.814 1125 TRACE nova return cctxt.call(context, 'ping', arg=arg_p) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2014-11-14 07:58:01.814 1125 TRACE nova retry=self.retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-11-14 07:58:01.814 1125 TRACE nova timeout=timeout, retry=retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2014-11-14 07:58:01.814 1125 TRACE nova retry=retry) 2014-11-14 07:58:01.814 1125 TRACE nova File "/usr/lib/python2.7/site-packages/oslo ... (more)

edit retag flag offensive close merge delete

add a comment

3 answers

Sort by » oldest newest most voted

answered 2014-11-17 14:51:54 -0500

federic0
36 ●3 ●4 ●7

i can't believe i spent so much time on this, but flushing iptables rules made nova-compute starting. thanks Herry for pointing me over there. now i need to understand which rule exactly avoid (or broke) rabbitmq connectivity.

edit flag offensive delete link

Comments

This shouldn't have happened.Do you have this line in your nova.conf in both compute and controller firewall_driver=nova.virt.firewall.NoopFirewallDriver also you have to set correct neutron firewall and it will applies the correct ip tables but flushing isn't the right thing to do

bishoy ( 2014-11-18 08:28:44 -0500 )edit

i didn't yet because if i follow the official installation guide the step where the firewall driver must be added comes later, configuration of compute/controller nodes is in chapter 5, neutron starts from chapter 6. so, no i don't have it yet.

federic0 ( 2014-11-18 10:17:16 -0500 )edit

moreover, there's another modification i have to apply any time i add a DB in mysql, GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'NOVA_DBPASS'; is ok, but i need to add it also for 'nova@controller' otherwise i get no access to DB; and this is valid for keystone, glance etc

federic0 ( 2014-11-18 10:19:20 -0500 )edit

anyway, adding the firewll_driver does not make nova-compute starting, becasue amqp protocol is always blocked by iptables, syn comes from compute node, but no ack comes back from controller on port 5672. where i can troubleshoot those iptables rules? which is the component that sets iptables rules?

federic0 ( 2014-11-18 14:21:17 -0500 )edit

add a comment

answered 2014-11-11 19:51:31 -0500

Eric Xie
111 ●14 ●16 ●22

You can try to start nova-compute from daemon to foreground. Follow this: 1. systemctl stop openstack-nova-compute.service 2. python /usr/bin/nova-compute --debug Then check if error exist. Wish help~

edit flag offensive delete link

Comments

OK thanks, i will try as soon as possible and let you know.

federic0 ( 2014-11-12 05:16:48 -0500 )edit

I also encounter this error. When I execute the command " python /usr/bin/nova-compute --debug " , the console hang up always. Anyone has any idea?

herry ( 2014-11-14 02:30:16 -0500 )edit

me too, check the logs under /var/log/nova and let me know if you see rabbitmq intermittent connectivity

federic0 ( 2014-11-15 01:29:04 -0500 )edit

the nova-compute.log always output the error : AMQP server on controller:5672 is unreachable. But I can ping the domain controller, telnet controller 5672 says "no route to host". I can telnet controller 5672 on controller node self. controller and compute node has multi nics and use suse centos7

herry ( 2014-11-15 20:36:23 -0500 )edit

While I use the iptables -F to clear the iptable at last,everything is ok. the openstack-nova-compute.service can be started

herry ( 2014-11-16 22:23:15 -0500 )edit

see more comments

answered 2014-11-11 17:03:57 -0500

dgalvao

83 ●3 ●4 ●10

updated 2014-11-12 14:58:44 -0500

Update: maybe this helps you.

The issue I was seeing was caused by the /var filesystem dropping below 1GB and rabbitmq blocking publishers. From the rabbitmq logs:

=INFO REPORT==== 12-Nov-2014::15:14:32 ===
Disk free space insufficient. Free bytes:696029184 Limit:1000000000

=WARNING REPORT==== 12-Nov-2014::15:14:32 ===
disk resource limit alarm set on node 'rabbit@cha-ops-xen-v04'.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************

I'm having a similar issue. I'm following that guide too and got the controller node running on a VM with keystone, glance and nova (minus compute) and all worked fine. However, on the compute node (running on a real server) I can't get the nova-compute service to start at all. It just hangs there with few helpful log messages even with the debug option on.

If you just run it as a service you will notice that it will timeout at about 90 seconds and try to start again and will just stay in that loop. For troubleshooting, I'm trying to start it manually: /usr/bin/nova-compute --debug --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log

You say "it won't start anymore". Did you ever get it to start successfully?

I've asked the same question a couple of hours ago, but it's waiting moderation. Hopefully we will have an answer soon.

edit flag offensive delete link

Comments

it started once! the installation guide let's you verify the part where you were working on, and at the end of chapter 5 "verify operation" was successful. then i moved ahead on neutron where i've got another issue (neutron agent-list on the controller node gives nothing) and i discovered this issue

federic0 ( 2014-11-12 05:16:09 -0500 )edit

unfortunatley i got no disk space issues.

federic0 ( 2014-11-14 01:16:38 -0500 )edit

add a comment

nova-compute won't start anymore

UPDATE:

3 answers

Comments

Comments

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

nova-compute won't start anymore edit

UPDATE:

3 answers

Comments

Comments

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

nova-compute won't start anymore