Ask Your Question
0

HEAT stack stuck on "Create In Progress"

asked 2020-04-10 04:37:32 -0500

Franjo gravatar image

Hi, I'm trying to deploy a kubernetes cluster via magnum. I'm creating the cluster as follows:

openstack coe cluster template create clustertemplate3 \
--coe kubernetes \
--server-type vm \
--image Fedora-Atomic-27-20180419.0.x86_64 \
--external-network provider \
--network-driver calico \
--volume-driver cinder \
--docker-volume-size 1 \
--docker-storage-driver cinder \
--dns-nameserver 8.8.8.8 \
--flavor m1.small \
--registry-enabled \
--labels cloud_provider_enabled=true

openstack coe cluster create \
--cluster-template clustertemplate3 \
--docker-volume-size 3 \
--keypair keypair1 \
--master-count 1 \
--node-count 1 \
--master-flavor m1.small \
--flavor m1.small \
--floating-ip-enabled \
cluster

The cluster status hangs in "CREATE_IN_PROGRESS" and eventually times out. There is an issue when creating the kube-master. The VM gets created fine, but I guess there's a problem with provisioning. I'm using Fedora-Atomic-27-20180419.0.x86_64.qcow2 for the image, because I had less luck with newer ones. Here is the full "journalctl" output of the kube-master VM: https://pastebin.com/VRkvPvA0

I'm guessing the issue lies with:

 Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: Authorization failed: Unable to establish connection to http://127.0.0.1:5000/v3/auth/tokens
Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: Source [heat] Unavailable.
Apr 10 09:18:02 cluster-kxo37xfwor3i-master-0.novalocal runc[1507]: /var/lib/os-collect-config/local-data not found. Skipping

...But 127.0.0.1:5000 isn't the IP I set in heat.conf. If I try curl-ing the controller IP address I see that it is reachable from the VM.

[root@cluster-kxo37xfwor3i-master-0 ~]# curl http://192.168.5.100:5000/v3/auth/tokens
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

In the VM,

/var/lib/os-collect-config/heat_local.json

and

/etc/os-collect-config.conf

both point to the correct IP of the openstack controller node.

[root@cluster-kxo37xfwor3i-master-0 ~]# cat /var/lib/os-collect-config/heat_local.json
{
 "os-collect-config": {
  "heat": {
   "password": "!WSa!NEmYAzVE@YBixypi5eZ3NpR80U8",
   "user_id": "7dba31ab737c4573b1b0fb48f82d9cc3",
   "region_name": null,
   "stack_id": "cluster-kxo37xfwor3i-kube_masters-q67abiuxjita-0-pbgqfoultwol/1335ffa0-d82e-4b1a-9c0b-874a97b8e7ba",
   "resource_name": "kube-master",
   "auth_url": "http://192.168.5.100:5000/v3/",
   "project_id": "0b3a89797867438daea494732195a714"
  },
  "collectors": [
   "ec2",
   "heat",
   "local"
  ]
 },
 "deployments": []
}[root@cluster-kxo37xfwor3i-master-0 ~]# 
[root@cluster-kxo37xfwor3i-master-0 ~]# cat /etc/os-collect-config.conf 
[DEFAULT]
command = os-refresh-config
collectors = ec2
collectors = heat
collectors = local


[heat]
auth_url = http://192.168.5.100:5000/v3/
user_id = 7dba31ab737c4573b1b0fb48f82d9cc3
password = !WSa!NEmYAzVE@YBixypi5eZ3NpR80U8
project_id = 0b3a89797867438daea494732195a714
stack_id = cluster-kxo37xfwor3i-kube_masters-q67abiuxjita-0-pbgqfoultwol/1335ffa0-d82e-4b1a-9c0b-874a97b8e7ba
resource_name = kube-master
region_name =

Here is my heat.conf

[DEFAULT]
heat_metadata_server_url = http://192.168.5.100:8000
heat_waitcondition_server_url = http://192.168.5.100:8000/v1/waitcondition
stack_user_domain_name = heat
stack_domain_admin = heat_domain_admin
stack_domain_admin_password = PASSWORD
transport_url = rabbit://guest:guest@192.168.5.100
[auth_password]
[clients]
endpoint_type = public
[clients_aodh]
[clients_barbican]
[clients_cinder]
[clients_designate]
[clients_glance]
[clients_heat]
[clients_keystone]
endpoint_type = public
auth_uri = http://192.168.5.100:5000
[clients_magnum]
endpoint_type = public
[clients_manila]
[clients_mistral]
[clients_monasca]
[clients_neutron]
[clients_nova]
[clients_octavia]
[clients_sahara]
[clients_senlin]
[clients_swift]
[clients_trove]
[clients_zaqar]
[cors]
[database]
connection = mysql+pymysql://heat:HEAT_DBPASS@192.168.5.100/heat
[ec2authtoken]
[eventlet_opts]
[healthcheck]
[heat_api]
[heat_api_cfn]
[heat_api_cloudwatch]
[noauth]
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_middleware]
[oslo_policy]
[paste_deploy]
[profiler]
[revision]
[ssl]
[keystone_authtoken]
www_authenticate_uri = http://192.168.5.100:5000
auth_url = http://192.168.5.100:5000
memcached_servers = 192.168.5.100:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = services
username = heat
password = PASSWORD
[trustee]
auth_type = password
auth_url = http://192.168.5.100:5000 ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2020-05-14 07:24:16 -0500

Franjo gravatar image

Right, so my problem wasn't with the heat config. It was keystone. Specifically the below line in /etc/keystone/keystone.conf

public_endpoint=http://127.0.0.1:5000

Heat just sends this to the VMs as the auth url, and since the VMs can't reach that that address the whole stack hangs. I just changed the above to the corresponding controller IP address that the VMs can reach:

public_endpoint=http://192.168.5.100:5000

After the change, everything worked.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2020-04-10 04:37:32 -0500

Seen: 178 times

Last updated: May 14