Ask Your Question

Magnum vm instance can ping controller ip but can't resolve it by hostname to notify heat of status

asked 2016-09-14 14:08:16 -0600

mitakauser gravatar image

updated 2016-09-15 13:18:44 -0600

Hi ,

We're currently implementing a magnum bay deployment for testing purposes. So far we've been able to create a bay successfully but not without a few quirks. I know that magnum calls the heat api to create the whole bay (master, minion, health monitor,lbaas,etc). I was getting curious as to why after 1 hour the bay status remained in "Create in Progress" and only one instance was created (baymodel config with 1 master and 1 minion) so I dove into the system logs (fedora 23) via journalctl and one thing caught my attention. This line in particular:

Aug 26 01:16:01 fe-tdlnn6z37o-0-j4pmkkqrwsjx-kube-master-mb5kf6rv4vzp.novalocal wc-notify[1580]: okcurl -i -X POST -H 'X-Auth-Token: gAAAAABXv5ZTcss3qrjgtVFM-f5zxrrEEC-Mx3Qbx7O_1fJUeqkkn0lM5omb9U3UvM-NBHS0pKKFcRJp0Apa7Zxz_MGBDssRdLUSA-e13hoEhBFOf-r3ZrXZ_4AegdbziOYSm6C23cYZZW_HU-pu2Fs6vpe4taFP_HaKOBHGdVA9wiaoq7qC99E' -H 'Content-Type: application/json' -H 'Accept: application/json' **http://controller:8004/v1/9d7a25b6e7a74bce9fc09f06a3b0e258/stacks/fedora-23-bay-76m7za4mekfi-kube_masters-mjtdlnn6z37o-0-j4pmkkqrwsjx/d558128b-96e3-4ddb-a8ba-f464995fd118/resources/master_wait_handle/signal** --data-binary '{"status": "SUCCESS"}'

In the magnum heat template directory (controller node) I found a script that makes this call, I'm assuming that this is to notify heat when the master node (and minion) creation was successful so heat can update its status back to cli/horizon. So from the instance I tried to ping controller first by ip address which was successful and then by hostname "controller" with output "unknown host controller". Only after I put in the entry in the /etc/hosts file (vm instance) the aforementioned call to the controller was able to notify heat of the master creation (same case as with the minions). My question is there a way to automate the name resolving for controller hostname without having to ssh into each node instance and modifying /etc/hosts?. Unless of course I'm missing something else here in which case any pointers would be greatly appreciated. Please also let me know if you need more information I'll be happy to provide it.

I followed this guide for installing magnum


edit retag flag offensive close merge delete


I got the same error as you. Have you solved the problem? Please let me know. Thanks!

JackLin gravatar imageJackLin ( 2016-09-29 00:32:50 -0600 )edit

Hi there, I'm configuring magnum in queens in centos 7 and I got stuck in cluster create in progress and even same status I'm experiencing where one master is created and no minion, thereafter when I log in for Kube-master instance I'm able to ping controller from the hostname.

juscloud gravatar imagejuscloud ( 2018-04-27 08:11:03 -0600 )edit

WHen i'm checking heat-engine logs i saw ""creating NoneResource "docker_volume_attach" Stack "kubernetes-cluster"" and magnum-conductor logs as "'Status for cluster 2 updated to CREATE_IN_PROGRESS (None) ""

juscloud gravatar imagejuscloud ( 2018-04-27 08:14:01 -0600 )edit

So, Sir, Pls help further what exactly I need to do for cluster create complete as even I made host entry in Kube-master instance of controller...any help would be highly appreciable.

juscloud gravatar imagejuscloud ( 2018-04-27 08:15:22 -0600 )edit

There seems to be major problems in the magnum. The user page, supposedly updated a week ago is full of bad links. plus the latest "fedora" kernels that are supposed to work, will NOT bring up a running Kube OR docker cluster. RExray seems to be throwing kernel panics, which in turn causes bad sys

hardcore-01 gravatar imagehardcore-01 ( 2018-06-23 21:09:40 -0600 )edit

2 answers

Sort by ยป oldest newest most voted

answered 2016-09-15 20:41:10 -0600

check your /etc/resolv.conf and make sure you can pull the record from the host with dig:

dig @dns.server

if you can do this, your host should be able to resolve it.

edit flag offensive delete link more

answered 2018-06-13 03:27:31 -0600

mago gravatar image


Using Magnum for Pike and ran into the same issue. Please find below some detail about the issue, and the solution(s) I found.

The kubernetes nodes (both master and minions) need to reach several controller endpoints. At least the public Keystone, Heat, and Magnum endpoints, maybe more. The k8s nodes in my environment are also not able to resolve the controller hostname, and therefore can't reach the necessary endpoints.

I see three solutions to this issue.

1 Configure a DNS

This is probably the cleanest way to resolve the issue, especially if you intend to go prod. Configure a DNS and edit your magnum template to use this DNS :

openstack coe cluster template create [...] --dns-nameserver <my_DNS_IP> [...]

2 Configure local hostname resolution on your k8s nodes

This can be done by automatically pushing a /etc/hosts file into the k8s nodes. I did that by modifying the Heat templates used by Magnum.

In my case, the templates used for the Masters and Minions were found in this directory :


I added a resource configure_etc_hosts and added it to the init scripts in both files :

  configure_etc_hosts:       <---------------------------- Add this whole block
    type: OS::Heat::SoftwareConfig
      group: ungrouped
      config: |
        echo "<my_controller_IP> controller" >> /etc/hosts     <---- Change <my_controller_IP> to your actual controller IP
    type: OS::Heat::MultipartMime
        - config: {get_resource: configure_etc_hosts}      <-------------- Add this line
        - config: {get_resource: install_openstack_ca}
        - config: {get_resource: disable_selinux}

3 Change the controller endpoints to use IP addresses

According to me, this is not ideal for prod environments but can be useful for small test clusters.

You can edit the controller endpoints so the public endpoints for Keystone, Heat, and Magnum (at least, maybe more) are using the controller's IP address instead of the controller's hostname. The command to run, for example :

openstack endpoint set --url "http://<my_controller_IP>:8004/v1/%(tenant_id)s" d8be3b40d0724d89b2baa6f2aa8014a4

I know this is an old thread, but as I hit the same issues and found a solution, I believe this might be helpful for anyone in the same situation as me in the future.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2016-09-14 14:08:16 -0600

Seen: 773 times

Last updated: Jun 13 '18