HEAT stack stuck on "Create In Progress"

asked 2020-04-10 04:37:32 -0600

Franjo gravatar image

Hi, I'm trying to deploy a kubernetes cluster via magnum. I'm creating the cluster as follows:

openstack coe cluster template create clustertemplate3 \
--coe kubernetes \
--server-type vm \
--image Fedora-Atomic-27-20180419.0.x86_64 \
--external-network provider \
--network-driver calico \
--volume-driver cinder \
--docker-volume-size 1 \
--docker-storage-driver cinder \
--dns-nameserver \
--flavor m1.small \
--registry-enabled \
--labels cloud_provider_enabled=true

openstack coe cluster create \
--cluster-template clustertemplate3 \
--docker-volume-size 3 \
--keypair keypair1 \
--master-count 1 \
--node-count 1 \
--master-flavor m1.small \
--flavor m1.small \
--floating-ip-enabled \

The cluster status hangs in "CREATE_IN_PROGRESS" and eventually times out. There is an issue when creating the kube-master. The VM gets created fine, but I guess there's a problem with provisioning. I'm using Fedora-Atomic-27-20180419.0.x86_64.qcow2 for the image, because I had less luck with newer ones. Here is the full "journalctl" output of the kube-master VM: https://pastebin.com/VRkvPvA0

I'm guessing the issue lies with:

 Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: Authorization failed: Unable to establish connection to
Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: Source [heat] Unavailable.
Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: /var/lib/os-collect-config/local-data not found. Skipping

...But isn't the IP I set in heat.conf. If I try curl-ing the controller IP address I see that it is reachable from the VM.

[root@cluster-kxo37xfwor3i-master-0 ~]# curl
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

In the VM,




both point to the correct IP of the openstack controller node.

[root@cluster-kxo37xfwor3i-master-0 ~]# cat /var/lib/os-collect-config/heat_local.json
 "os-collect-config": {
  "heat": {
   "password": "!WSa!NEmYAzVE@YBixypi5eZ3NpR80U8",
   "user_id": "7dba31ab737c4573b1b0fb48f82d9cc3",
   "region_name": null,
   "stack_id": "cluster-kxo37xfwor3i-kube_masters-q67abiuxjita-0-pbgqfoultwol/1335ffa0-d82e-4b1a-9c0b-874a97b8e7ba",
   "resource_name": "kube-master",
   "auth_url": "",
   "project_id": "0b3a89797867438daea494732195a714"
  "collectors": [
 "deployments": []
}[root@cluster-kxo37xfwor3i-master-0 ~]# 
[root@cluster-kxo37xfwor3i-master-0 ~]# cat /etc/os-collect-config.conf 
command = os-refresh-config
collectors = ec2
collectors = heat
collectors = local

auth_url =
user_id = 7dba31ab737c4573b1b0fb48f82d9cc3
password = !WSa!NEmYAzVE@YBixypi5eZ3NpR80U8
project_id = 0b3a89797867438daea494732195a714
stack_id = cluster-kxo37xfwor3i-kube_masters-q67abiuxjita-0-pbgqfoultwol/1335ffa0-d82e-4b1a-9c0b-874a97b8e7ba
resource_name = kube-master
region_name =

Here is my heat.conf

heat_metadata_server_url =
heat_waitcondition_server_url =
stack_user_domain_name = heat
stack_domain_admin = heat_domain_admin
stack_domain_admin_password = PASSWORD
transport_url = rabbit://guest:guest@
endpoint_type = public
endpoint_type = public
auth_uri =
endpoint_type = public
connection = mysql+pymysql://heat:HEAT_DBPASS@
www_authenticate_uri =
auth_url =
memcached_servers =
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = services
username = heat
password = PASSWORD
auth_type = password
auth_url = ...
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2020-05-14 07:24:16 -0600

Franjo gravatar image

Right, so my problem wasn't with the heat config. It was keystone. Specifically the below line in /etc/keystone/keystone.conf


Heat just sends this to the VMs as the auth url, and since the VMs can't reach that that address the whole stack hangs. I just changed the above to the corresponding controller IP address that the VMs can reach:


After the change, everything worked.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2020-04-10 04:37:32 -0600

Seen: 504 times

Last updated: May 14 '20