Ask Your Question
1

VM stucking at Spawning

asked 2014-05-09 14:20:18 -0500

Sonia gravatar image

updated 2014-05-12 18:36:55 -0500

9lives gravatar image

Hi

I have installed openstack using devstack.When i'm booting VM from my openstack dashboard onto my compute node,everytime it stucks at spawning state. Moreover there is no error in nova-compute service. What can be the issue. Please help regarding this

Thanks Sonia

edit retag flag offensive close merge delete

Comments

Could u paste the local.conf and nova-scheduler.log here for us to take a look?

9lives gravatar image9lives ( 2014-05-09 19:16:10 -0500 )edit

Here is my local.conf..

[[local|localrc]] LOGFILE=stack.sh.log #OFFLINE=true RECLONE=yes disable_all_services enable_service n-cpu n-novnc nova neutron rabbit mysql q-agt #enable_service neutron nova n-cpu quantum q-agt n-novnc qpid

HOST_NAME= HOST_IP=192.168.8.49 SERVICE_HOST_NAME=shiv-Vostro-2520 SERVICE_HOST=192.168.8.52 FLOATING_RANGE=192.168.8.0/24

VNCSERVER_PROXYCLIENT_ADDRESS=192.168.8.49 VNCSERVER_LISTEN=0.0.0.0

Q_HOST=$SERVICE_HOST MYSQL_HOST=$SERVICE_HOST RABBIT_HOST=$SERVICE_HOST GLANCE_HOSTPORT=$SERVICE_HOST:9292 KEYSTONE_AUTH_HOST=$SERVICE_HOST KEYSTONE_SERVICE_HOST=$SERVICE_HOST

MYSQL_PASSWORD=mysql RABBIT_PASSWORD=rabbit QPID_PASSWORD=rabbit SERVICE_TOKEN=service SERVICE_PASSWORD=admin ADMIN_PASSWORD=admin

LOGFILE=/opt/stack/logs/stack.sh.log VERBOSE=True LOG_COLOR=False SCREEN_LOGDIR=/opt/stack/logs EXTRA_OPTS=(scheduler_default_filters=AllHostsFilter)

Since it is my compute,there are no schedular logs in it.

Thanks

Sonia gravatar imageSonia ( 2014-05-10 07:18:25 -0500 )edit

looks like u are deploy in a multinode without using neutron could u paste the local.conf in controller node as well as the scheduler log?

9lives gravatar image9lives ( 2014-05-10 20:07:31 -0500 )edit

he is my controller local.conf.. [[local|localrc]] LOGFILE=stack.sh.log SCREEN_LOGDIR=/opt/stack/data/log LOG_COLOR=False #OFFLINE=True RECLONE=yes

enable_service rabbit disable_service qpid

enable_service n-cond disable_service n-net enable_service q-svc enable_service q-agt enable_service q-dhcp enable_service q-l3 enable_service q-meta enable_service neutron

FLAT_INTERFACE=eth1 FIXED_NETWORK_SIZE=254 FLOATING_RANGE=192.168.6.0/24

HOST_IP=192.168.6.107 HOST_NAME=shiv-Vostro-2520 SERVICE_HOST_NAME=${HOST_NAME} SERVICE_HOST=192.168.6.107

MYSQL_HOST=$SERVICE_HOST RABBIT_HOST=$SERVICE_HOST GLANCE_HOSTPORT=$SERVICE_HOST:9292 KEYSTONE_AUTH_HOST=$SERVICE_HOST KEYSTONE_SERVICE_HOST=$SERVICE_HOST

MYSQL_PASSWORD=mysql RABBIT_PASSWORD=rabbit QPID_PASSWORD=rabbit SERVICE_TOKEN=service SERVICE_PASSWORD=admin ADMIN_PASSWORD=admin

Sonia gravatar imageSonia ( 2014-05-10 23:59:48 -0500 )edit

This is not enough info for the troubleshooting, cloud u paste the whole nova-scheduler.log in the folder SCREEN_LOGDIR (i.e./opt/stack/data/log) started with screen-*, this is very important log when instance provision stucked.:)

9lives gravatar image9lives ( 2014-05-11 01:46:24 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-05-11 18:19:25 -0500

9lives gravatar image

updated 2014-05-12 18:34:30 -0500

Well from the scheduler log , the host filter seemed picked the only available compute node is t4240-ubuntu1310,two things need to be checked:

  1. controller node might has nova-compute service enabled?
  2. Can the compute node connect to the rabbitmq or just check the nov-compute log in compute node.

    ... Host filter forcing available hosts to t4240-ubuntu1310 2014-05-11 18:48:51.375 DEBUG nova.scheduler.filter_scheduler [req-e36b289c-510c-4f47-9500-73f9a91c5f0e admin admin] Filtered [(t4240-ubuntu1310, t4240-ubuntu1310) ram:4352 disk:97280 io_ops:2 instances:2] _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:331 2014-05-11 18:48:51.376 DEBUG nova.scheduler.filter_scheduler [req-e36b289c-510c-4f47-9500-73f9a91c5f0e admin admin] Weighed [WeighedHost [host: t4240-ubuntu1310, weight: 1.0]] _sc ... If case 2 is true, just remove the nova-compute service from controller node, next time you will create instance in the compute node only.

    ... DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70 2014-05-10 03:43:34.810 1651 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420 2014-05-10 03:43:34.811 INFO nova.openstack.common.rpc.common [req-f58a2c63-11b9-4b73-91c2-9495e5831099 None None] Connected to AMQP server on 192.168.8.96:5672 2014-05-10 03:43:34.816 DEBUG nova.openstack.common.rpc.amqp [req-f58a2c63-11b9-4b73-91c2-9495e5831099 None None] Pool creating new connection create /opt/stack/nova/nova/openstack/common/rpc/amqp.py:75 2014-05-10 03:43:34.836 1651 DEBUG amqp [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/' , u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'3.1.3'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'en_US'] _start /usr/lib/python2.7/dist-packages/amqp/connection.py:706 2014-05-10 03:43:34.838 1651 DEBUG amqp [-] Open OK! _open_ok /usr/lib/python2.7/dist-packages/amqp/connection.py:592 2014-05-10 03:43:34.838 1651 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70 2014-05-10 03:43:34.839 1651 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420 2014-05-10 03:43:34.840 INFO nova.openstack.common.rpc.common [req-f58a2c63-11b9-4b73-91c2-9495e5831099 None None] ...

I guess you must use rabbitmq ha or something? The issue should be lies in the nova-compute node cannot update instance state via nova-conductor service in mysql db. Please make sure the nova-compute and nova-conductor service connected to the same rabbitmq server.

Hope that helps.

Vic

edit flag offensive delete link more

Comments

The rabbit on the compute node is running.I removed the nova-compute service on the controller node and still the VM is stucking at spawning state.Below are the nova-compute logs of compute node

cd /opt/stack/nova && sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || echo "n-cpu failed to start" | tee "/opt/stack/status/stack/n-cpu.failure" 2014-05-10 03:43:33.962 1651 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /opt/stack/nova/nova/servicegroup/api.py:62 2014-05-10 03:43:34.223 1651 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2014-05-10 03:43:34.445 1651 DEBUG stevedore.extension [-] found extension EntryPoint.parse('file = nova.image.download.file') _load_plugins /usr/local/lib/python2.7/dist-packages/stevedore/extension.py:129 2014-05-10 03:43:34.501 1651 DEBUG stevedore.extension [-] found extension EntryPoint.parse ...(more)

Sonia gravatar imageSonia ( 2014-05-12 00:46:08 -0500 )edit

Also I'm not able to boot any VM manually on compute node using KVM utility. Can this be the reason for the abbove issue.

Sonia gravatar imageSonia ( 2014-05-12 01:28:04 -0500 )edit

Need the compute node nova.conf and controller node scheduler.log could u past them in paste.openstack.org? Besides check the nova-conductor log make sure it is running with no error.

Another try : restart rabbit mq server and nova related service on controller node then restart nova-compute service on compute node and try to create instance again.

9lives gravatar image9lives ( 2014-05-12 01:48:05 -0500 )edit

Here are the logs of nova-cond at the controller side..

removed from py-amqp v1.5.0. warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED)) 2014-05-12 19:04:32.826 30311 WARNING nova.context [-] Arguments dropped when creating context: {'quota_class': None} /usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed from py-amqp v1.5.0. warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED)) /usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed from py-amqp v1.5.0. warn(VDeprecationWarning(EXCHANGE_AUTODELETE_DEPRECATED)) 2014-05-12 19:04:33.228 30310 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call sleeping for 60.00 seconds _inner /opt/stack/nova/nova/openstack/common/loopingcall.py:130 /usr/local/lib/python2.7/dist-packages/amqp/channel.py:616: VDeprecationWarning: The auto_delete flag for exchanges has been deprecated and will be removed from ...(more)

Sonia gravatar imageSonia ( 2014-05-12 08:42:34 -0500 )edit

Hi

I retried my setup and I'm able to see the VM interface onto my compute node but still the VM is stucking at spawning state

Nova-compute logs at the compute node are repeating continuesly.Below are the logs.

-05-16 05:34:19.503 26935 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._poll_volume_usage run_periodic_tasks /opt/stack/nova/nova/openstack/common/periodic_task.py:176^M 2014-05-16 05:34:19.504 26935 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager._instance_usage_audit run_periodic_tasks /opt/stack/nova/nova/openstack/common/periodic_task.py:176^M 2014-05-16 05:34:19.504 26935 DEBUG nova.openstack.common.periodic_task [-] Running periodic task ComputeManager.update_available_resource run_periodic_tasks /opt/stack/nova/nova/openstack/common/periodic_task.py:176^M 2014-05-16 05:34:19.505 26935 DEBUG nova.openstack.common.lockutils [-] Got semaphore "compute_resources" lock /opt/stack/nova/nova/openstack/common/lockutils.py:166^M 2014-05-16 05:34:19 ...(more)

Sonia gravatar imageSonia ( 2014-05-16 05:52:59 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-05-09 14:20:18 -0500

Seen: 1,344 times

Last updated: May 12 '14