Ask Your Question
2

nova-scheduler driver setting instance to ERROR state

asked 2013-05-16 10:08:26 -0500

henrik16 gravatar image

updated 2013-05-17 05:43:48 -0500

My instances are going to error state and No State in Power State... so I wen to logs and I foud this in the nova-scheduler.log:

2013-05-16 16:05:52.278 WARNING nova.scheduler.driver [req-41c57487-d1fd-44fc-bf2e-49fe64242a5e 19438051351b45f99cf8b78843d383bd e29ae90e318c4e6786547f6a4bda2871] [instance: 8eea4d0d-4b36-4965-a955-4112263a85ad] Setting instance to ERROR state.

Complete log here

$ nova-manage service list

    Binary           Host                                 Zone             Status     State Updated_At
nova-cert        cloud                                internal         enabled    :-)   2013-05-17 10:41:03
nova-consoleauth cloud                                internal         enabled    :-)   2013-05-17 10:41:01
nova-scheduler   cloud                                internal         enabled    :-)   2013-05-17 10:41:05
nova-conductor   cloud                                internal         enabled    :-)   2013-05-17 10:41:03
nova-compute     c01                                  nova             enabled    XXX   2013-05-15 15:40:44
nova-compute     c02                                  nova             enabled    XXX   2013-05-15 15:42:31

As you can see nova-compute services are with XXX state and when I run service nova-compute status I get: nova-compute stop/waiting the nova-compute service is not starting:

root@c01:~# service nova-compute restart
stop: Unknown instance:
nova-compute start/running, process 16168

Anyone know what is going on?

edit retag flag offensive close merge delete

Comments

I have the same problem, I am running controller on a debian jessie VM and the compute/network node on an UBUNTU 14.04?

Moe gravatar imageMoe ( 2014-09-28 02:19:29 -0500 )edit

6 answers

Sort by ยป oldest newest most voted
3

answered 2013-05-21 05:15:50 -0500

darragh-oreilly gravatar image

So your /var/log/upstart/nova-compute.log has: oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files: /etc/nova/nova.conf,/etc/nova/nova-compute.conf

This is probably because user nova does not have permissions to read them (ls -l /etc/nova/). Try 'chown nova:nova /etc/nova/*'

edit flag offensive delete link more

Comments

You are right nova user is not the owner of the nova files... I've changed the owner for all files and now? Reboot? Thanks!

henrik16 gravatar imagehenrik16 ( 2013-05-21 05:33:21 -0500 )edit
1

'service nova-compute start' should be enough.

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-05-21 05:58:53 -0500 )edit

root@c01:~# service nova-compute start nova-compute start/running, process 3950 root@c01:~# service nova-compute status nova-compute stop/waiting

I don't think so... :s

henrik16 gravatar imagehenrik16 ( 2013-05-21 06:00:23 -0500 )edit

so check the logs...

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-05-21 06:10:04 -0500 )edit

upstart/nova-compute.log stays the same... and nova-compute.log is clear... do I have to check more logs?

henrik16 gravatar imagehenrik16 ( 2013-05-21 06:13:40 -0500 )edit
1

answered 2013-08-02 05:44:36 -0500

sph gravatar image

For me the problem was that rabbitmq has a disk free threshold limit. If you disk space goes below that threshold, rabbitmq will block connections to queues. You can check these limits via rabbitmq's mgmt console by doing

/usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management

service rabbitmq-server restart

http://your_controller:5672

edit flag offensive delete link more

Comments

Had the same issue due to a small /var partition. You can check /var/log/rabbit@hostname.log for the following line =INFO REPORT==== 26-Nov-2013::09:37:23 === Disk free space insufficient. Free bytes:750694400 Limit:1000000000 Hard to figure out since new lines were generated. Thanks!

BregtBeer gravatar imageBregtBeer ( 2013-11-26 03:18:26 -0500 )edit
1

answered 2013-05-30 05:40:19 -0500

siserte gravatar image

updated 2013-05-30 05:42:04 -0500

Hi, I had the same problem, and I don't know why it is happenning. It seems to be that the launcher is not working how we expected.

I tried with executing nova-compute directly by using:

# cd /usr/bin

# ./nova-compute

And it worked for me.

Regards!

edit flag offensive delete link more
1

answered 2013-05-18 22:14:49 -0500

cfb gravatar image

It looks like your nova-compute service keeps dieing. Are there any errors or stack traces in /var/log/nova/nova-compute.log on each compute node?

edit flag offensive delete link more

Comments

Hi cfb thanks for your answer and sorry to all of you to take so long to reply! No in both nova-compute nodes there are nothing in the log.... clear... :/

henrik16 gravatar imagehenrik16 ( 2013-05-20 03:14:10 -0500 )edit
1

answered 2013-05-18 14:56:20 -0500

Attila Soki gravatar image

Hi Henrik,

The system is out of sync for the time. Check the NTP setting for the nodes or

nova-conductor cloud internal enabled :-) 2013-05-17 10:41:03 nova-compute c01 nova enabled XXX 2013-05-15 15:40:44 nova-compute c02 nova enabled XXX 2013-05-15 15:42:31

Then restart the nodes-*.

edit flag offensive delete link more

Comments

Hi Attila Soki, the problem now is that when I run nova-manage service list in cloud I only see three services, nova-cert, nova-consoleauth and nova-scheduler ! In the compute node 1 I see nova-conductor nova-compute with XXX state, in compute node 2 there are no services showing ! :/

henrik16 gravatar imagehenrik16 ( 2013-05-20 03:18:24 -0500 )edit

yes, ntp time sync solves my problem as well. Basically my worker nodes somehow got rebooted and they were out of sync with controller node. I am using a Folsom release.

changbl gravatar imagechangbl ( 2013-08-15 20:17:23 -0500 )edit
2

answered 2013-05-16 15:05:46 -0500

Ashokb gravatar image

Make sure your Nova-conductor and and Nova-compute services runs properly. nova-manager service list will show a :-) symbol if both are OK. if there are issues it will show as XXX.

Also make sure you have enough resources in the system like CPU and memory to launch the VM's

-Ashok

edit flag offensive delete link more

Comments

Thanks for your answer Ashok but it was running normaly before I try to put live-migrations working... the services, I have two nova-compute that are with XXX state :/ How can I solve that? Thanks...

henrik16 gravatar imagehenrik16 ( 2013-05-17 04:08:13 -0500 )edit

Obviously check nova-compute.log on the compute nodes. Are the node's clocks in sync?

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-05-17 07:51:01 -0500 )edit

Yes they're in sync.... the log has nothing in there ... :/

henrik16 gravatar imagehenrik16 ( 2013-05-17 08:05:33 -0500 )edit

Just to confirm - nova-compute.log is 0 bytes? If so then the command line used in its startup script must be wrong or maybe nova.conf. If nova-compute.log >0 then please post the last page or two to pastebin or somewhere. Where it is stopping might provide a clue even if it is not an clear ERROR.

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-05-20 04:11:57 -0500 )edit

-rw-r--r-- 1 109 116 0 May 16 06:31 nova-compute.log

henrik16 gravatar imagehenrik16 ( 2013-05-20 04:18:04 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

Stats

Asked: 2013-05-16 10:08:26 -0500

Seen: 8,692 times

Last updated: Aug 02 '13