Hi,

Wonder if the community could help me, I have an openstack enviroment bult as following but can not boot VM's, they are stuck in the scheduling state.(Please let me know if I need to provide more info, or there is a troubleshooting process that you could suggest)

I have 3 KVM hosts running OPENSTACk management services(Each Openstack component Replicated 3 times for HA) - VM1,2,3-Kloud-Controller1,2,3 running Cinder,Glance,NOVA,Neutron API's, these VM's are attached to a CEPH backend which stores the images and volumes for Glance and Cinder. 3 VM's running the rabbit messaging queue, 2 Mysql Vm's running Galera - for actvie-actve 3 VM's running neutron nodes - with ML2 plugin.

There is an CEPH cluster - with 3 Monitors and OSD daemons on each of the 3 nodes with DAs's attached to each host.CEPH cluster is in a healthy state and I have previously being able to boot virtual machines.

When I do a nova agent-list - it returns nothing :

root@kloud-controller1:~# nova agent-list
+----------+------------+----+--------------+---------+---------+-----+
| Agent_id | Hypervisor | OS | Architecture | Version | Md5hash | Url |
+----------+------------+----+--------------+---------+---------+-----+
+----------+------------+----+--------------+---------+---------+-----+

if I do a nova-manage - it returns services are fine:

root@kloud-controller1:~# nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary           Host                                 Zone             Status     State Updated_At
nova-cert        kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:27
nova-conductor   kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:27
nova-consoleauth kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:25
nova-scheduler   kloud-controller2                    internal         enabled    :-)   2015-06-08 09:25:28
nova-consoleauth kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:27
nova-cert        kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:27
nova-scheduler   kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:29
nova-conductor   kloud-controller1                    internal         enabled    :-)   2015-06-08 09:25:34
nova-consoleauth kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:32
nova-scheduler   kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:24
nova-cert        kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:33
nova-conductor   kloud-controller3                    internal         enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute1                       nova             enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute3                       nova             enabled    :-)   2015-06-08 09:25:25
nova-compute     kloud-compute4                       nova             enabled    :-)   2015-06-08 09:25:31
nova-compute     kloud-compute2                       nova             enabled    :-)   2015-06-08 09:25:

cat openrc.sh

export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=xxxxxx
export OS_AUTH_URL=http://172.x.x.x:35357/v3
export OS_IMAGE_API_VERSION=2
export OS_VOLUME_API_VERSION=2

keystone --debug catalog

root@kloud-controller1:~# keystone --debug catalog
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.x.x.x:35357/v3/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.x.x.x
DEBUG:urllib3.connectionpool:"POST /v3/tokens HTTP/1.1" 404 93
DEBUG:keystoneclient.session:Request returned failure status: 404
Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-b03f47bc-23cb-4383-9671-2ce1279b6738)

I have turned off all redundant VM's so that I have just one instance of each Openstack service and checked the nova scheduler log and cinder scheduler log.

root@kloud-controller1:~# tail -5 /var/log/nova/nova-scheduler.log
2015-06-08 10:37:39.746 1120 INFO nova.scheduler.host_manager [req-4721db32-89ac-47ad-9c54-ad37879572dc - - - - -] Successfully synced instances from host 'kloud-compute4 ...