Ask Your Question
1

nova compute keep going down, negative timestamp difference

asked 2015-03-10 13:46:17 -0500

dsrini gravatar image

updated 2015-03-10 20:39:21 -0500

smaffulli gravatar image

HI

I am using devstack to install icehouse and I have a single box setup (nova/neutron on the same box). Recently I have seen that the nova-compute service keep going down and coming back up.

If you check nova service-list, we find the update_at timestamp decreases and compute is determined DOWN.

2015-03-10T18:32:03.000000 -> 2015-03-10T18:28:30.000000

Please let me know what should be done, this is really affecting us.

Regards, Dev

stack@A1-24849:/opt/stack/devstack$ nova service-list
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-conductor   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:11.000000 | -               |
| **2  | nova-compute     | A1-24849 | nova     | enabled | up    | 2015-03-10T18:32:03.000000 | -**               |
| 3  | nova-cert        | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:05.000000 | -               |
| 4  | nova-scheduler   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:08.000000 | -               |
| 5  | nova-consoleauth | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:07.000000 | -               |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
stack@A1-24849:/opt/stack/devstack$ nova service-list
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-conductor   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:11.000000 | -               |
| **2  | nova-compute     | A1-24849 | nova     | enabled | down  | 2015-03-10T18:28:30.000000 | -**               |
| 3  | nova-cert        | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:05.000000 | -               |
| 4  | nova-scheduler   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:08.000000 | -               |
| 5  | nova-consoleauth | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:07.000000 | -               |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
stack@A1-24849:/opt/stack/devstack$ nova service-list
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-conductor   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:11.000000 | -               |
**| 2  | nova-compute     | A1-24849 | nova     | enabled | down  | 2015-03-10T18:28:30.000000 | -**               |
| 3  | nova-cert        | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:15.000000 | -               |
| 4  | nova-scheduler   | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:08.000000 | -               |
| 5  | nova-consoleauth | A1-24849 | internal | enabled | up    | 2015-03-10T18:32:07.000000 | -               |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

Few more details :

/opt/stack/logs/screen-n-api.2015-03-10-111953.log:2015-03-10 11:22:48.351 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-api.log:2015-03-10 11:22:48.351 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cauth.2015-03-10-111953.log:2015-03-10 11:23:31.090 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cauth.log:2015-03-10 11:23:31.090 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cond.2015-03-10-111953.log:2015-03-10 11:23:15.779 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cond.log:2015-03-10 11:23:15.779 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cpu.2015-03-10-111953.log:2015-03-10 11:23:12.718 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-cpu.log:2015-03-10 11:23:12.718 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs/screen-n-crt.2015-03-10-111953.log:2015-03-10 11:23:18.873 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
/opt/stack/logs ...
(more)
edit retag flag offensive close merge delete

Comments

Awesome, but the box i am using is fairly powerful and the CPUs are only 12-15% used up. So doubt if they are overloaded.

dsrini gravatar imagedsrini ( 2015-03-11 13:18:27 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
1

answered 2015-03-11 13:18:43 -0500

dsrini gravatar image

Awesome, but the box i am using is fairly powerful and the CPUs are only 12-15% used up. So doubt if they are overloaded.

edit flag offensive delete link more
0

answered 2015-03-10 22:32:05 -0500

I would say its ntp related but since its on the same box as conductor its hard to point at ntp.

Is your host overloaded and nova may not have enough CPU / Mem to complete the operation?

edit flag offensive delete link more

Comments

You should monitor and see next time it goes down and then look at nova-compute.logs and nova-conductor.logs. If you can paste them next time it happens it would be good.

sfcloudman gravatar imagesfcloudman ( 2015-03-11 17:43:37 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-03-10 13:46:17 -0500

Seen: 573 times

Last updated: Mar 10 '15