Ask Your Question
0

OPENSTACK KILO - VM's stuck in scheduling state

asked 2015-06-08 04:41:46 -0500

kildarejoe gravatar image

updated 2015-06-08 15:42:09 -0500

rbowen gravatar image

Hi,

Wonder if the community could help me, I have an openstack enviroment bult as following but can not boot VM's, they are stuck in the scheduling state.(Please let me know if I need to provide more info, or there is a troubleshooting process that you could suggest)

I have 3 KVM hosts running OPENSTACk management services(Each Openstack component Replicated 3 times for HA) - VM1,2,3-Kloud-Controller1,2,3 running Cinder,Glance,NOVA,Neutron API's, these VM's are attached to a CEPH backend which stores the images and volumes for Glance and Cinder. 3 VM's running the rabbit messaging queue, 2 Mysql Vm's running Galera - for actvie-actve 3 VM's running neutron nodes - with ML2 plugin.

There is an CEPH cluster - with 3 Monitors and OSD daemons on each of the 3 nodes with DAs's attached to each host.CEPH cluster is in a healthy state and I have previously being able to boot virtual machines.

When I do a nova agent-list - it returns nothing :

root@kloud-controller1:~# nova agent-list
+----------+------------+----+--------------+---------+---------+-----+
| Agent_id | Hypervisor | OS | Architecture | Version | Md5hash | Url |
+----------+------------+----+--------------+---------+---------+-----+
+----------+------------+----+--------------+---------+---------+-----+

if I do a nova-manage - it returns services are fine:

root@kloud-controller1:~# nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary           Host                                 Zone             Status     State Updated_At
nova-cert        kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:27
nova-conductor   kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:27
nova-consoleauth kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:25
nova-scheduler   kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:28
nova-consoleauth kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:27
nova-cert        kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:27
nova-scheduler   kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:29
nova-conductor   kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:34
nova-consoleauth kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:32
nova-scheduler   kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:24
nova-cert        kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:33
nova-conductor   kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute1                       nova             enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute3                       nova             enabled    :-)   2015-06-08 09:25:25
nova-compute     kloud-compute4                       nova             enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute2                       nova             enabled    :-)   2015-06-08 09:25:

cat openrc.sh

export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=xxxxxx
export OS_AUTH_URL=http://172.x.x.x:35357/v3
export OS_IMAGE_API_VERSION=2
export OS_VOLUME_API_VERSION=2

keystone --debug catalog

root@kloud-controller1:~# keystone --debug catalog
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.x.x.x:35357/v3/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.x.x.x
DEBUG:urllib3.connectionpool:"POST /v3/tokens HTTP/1.1" 404 93
DEBUG:keystoneclient.session:Request returned failure status: 404
Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-b03f47bc-23cb-4383-9671-2ce1279b6738)

I have turned off all redundant VM's so that I have just one instance of each Openstack service and checked the nova scheduler log and cinder scheduler log.

root@kloud-controller1:~# tail -5 /var/log/nova/nova-scheduler.log
2015-06-08 10:37:39.746 1120 INFO nova.scheduler.host_manager [req-4721db32-89ac-47ad-9c54-ad37879572dc - - - - -] Successfully synced instances from host 'kloud-compute4 ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2016-06-08 04:32:44 -0500

kildarejoe gravatar image

Maybe an problem with keystone ->

Verify correct functionality by request auth token and service catalog:

curl -i \ -H "Content-Type: application/json" \ -d ' { "auth": { "identity": { "methods": ["password"], "password": { "user": { "name": "admin", "domain": { "id": "default" }, "password": "Your_Admin_Password" } } } } }' \ http://HA_PRoxy_VIP_Controller_IP:353... ; echo

Just substitute in your admin password and VIP/controller IP, is that 172.x.x.x an VIP or controller IP?

edit flag offensive delete link more
0

answered 2018-08-31 14:39:23 -0500

vgdub gravatar image

Were you able to resovle this error ? I am also getting the same in Openstack kilo version. please let me know.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-06-08 04:41:46 -0500

Seen: 786 times

Last updated: Jun 08 '16