Ask Your Question
2

weird problem that nova-compute's state cannot be updated

asked 2013-12-02 22:34:00 -0500

tsjsdbd gravatar image

the state of all nova component:

root@controlH:/etc/neutron# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-cert        controlH                             internal         enabled    :-)   2013-12-03 16:33:27
nova-conductor   controlH                             internal         enabled    :-)   2013-12-03 16:33:27
nova-consoleauth controlH                             internal         enabled    :-)   2013-12-03 16:33:28
nova-scheduler   controlH                             internal         enabled    :-)   2013-12-03 16:33:28
nova-compute     computerH2                           nova             enabled    XXX   2013-11-30 10:49:17
nova-compute     havana13                             nova             enabled    XXX   2013-11-30 10:36:47

that i find the nova-compute cannot update it's state, evenif it is start very well (from log, i guess)

every component of openstack works fine, because each command executes well. !! Note that commands which read from mysql works find, but commands which will write to mysql will hang at:

 File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)

Here is the log of nova-compute:

2013-12-03 11:47:57.949 6880 TRACE stevedore.extension
2013-12-03 11:47:57.951 6880 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2013-12-03 11:47:58.006 6880 DEBUG nova.openstack.common.rpc.amqp [req-62f9a5ae-c484-49c6-ab38-ec710b49a8c7 None None] Making synchronous call on conductor ... multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:553
2013-12-03 11:47:58.007 6880 DEBUG nova.openstack.common.rpc.amqp [req-62f9a5ae-c484-49c6-ab38-ec710b49a8c7 None None] MSG_ID is 3d923045f28145918ac4042697f03e7b multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:556
2013-12-03 11:47:58.007 6880 DEBUG nova.openstack.common.rpc.amqp [req-62f9a5ae-c484-49c6-ab38-ec710b49a8c7 None None] UNIQUE_ID is da47ee4417524034a8b74a244339f7b5. _add_unique_id /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:341
2013-12-03 11:47:58.016 6880 DEBUG amqp [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'3.0.2'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] _start /usr/lib/python2.7/dist-packages/amqp/connection.py:706
2013-12-03 11:47:58.018 6880 DEBUG amqp [-] Open OK! _open_ok /usr/lib/python2.7/dist-packages/amqp/connection.py:592
2013-12-03 11:47:58.018 6880 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
2013-12-03 11:47:58.019 6880 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420
2013-12-03 11:47:58.020 6880 INFO nova.openstack.common.rpc.common [req-62f9a5ae-c484-49c6-ab38-ec710b49a8c7 None None] Connected to AMQP server on 128.6.3.103:5672
2013-12-03 11:47:58.023 6880 DEBUG nova.openstack.common.rpc.amqp [req-62f9a5ae-c484-49c6-ab38-ec710b49a8c7 None None] Pool creating new connection create /usr/lib/python2.7 ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
2

answered 2013-12-04 02:50:37 -0500

tsjsdbd gravatar image

after 3 days investigate, the criminal is the rabbitmq, for the disk limited reached, so it bolcking all flows. resolve it by reset rabbitmq's config, database dir. btw, it is really hard to find the problem while almost component's log were pretty well.

only rabbitmq has one single warning:

=INFO REPORT==== 4-Dec-2013::15:20:59 ===
Disk free space insufficient. Free bytes:311922688 Limit:1000000000

=WARNING REPORT==== 4-Dec-2013::15:20:59 ===
disk resource limit alarm set on node rabbit@controlH
edit flag offensive delete link more

Comments

You, sir, are a life saver and a genius. Thank you. Almost burned my cookies, but you saved them.

Andrew Kinney gravatar imageAndrew Kinney ( 2014-03-17 23:59:56 -0500 )edit

kudos from me as well, just had the same problem. It's a bit ridiculous that every other log/status/etc was fine! very misleading :)

Alexei gravatar imageAlexei ( 2014-05-12 11:10:58 -0500 )edit
0

answered 2014-04-24 06:37:54 -0500

rnelan gravatar image

updated 2014-04-24 06:38:53 -0500

Hello! Thanks for the answer! The only problem I have is not understanding how to "reset rabbitmq's config, database dir.". could you please explain it to me?

thanks!

nova-compute http://node-5.thecloudsolution.net nova enabled down 3 days, 22 hours nova-compute http://node-11.thecloudsolution.net nova enabled down 2 days, 18 hours

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

Stats

Asked: 2013-12-02 22:34:00 -0500

Seen: 3,010 times

Last updated: Apr 24 '14