Ask Your Question
0

Swarm cluster-create fails

asked 2016-11-24 07:39:30 -0500

Mutty Putty gravatar image

updated 2017-01-10 08:09:43 -0500

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on CentOS 7. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

 [hpchost1@controller ~]$ magnum cluster-show swarm-cluster_test
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                      |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                              |
| cluster_template_id | 15bdcb55-a334-4699-ad86-f98ffa338589                                                                                                                                                                                       |
| uuid                | 3e6c1f12-e9e9-4fbd-8262-d32d89aa7c3e                                                                                                                                                                                       |
| stack_id            | 89003fb7-a094-4ed8-b744-00121e8cd577                                                                                                                                                                                       |
| status_reason       | Timed out                                                                                                                                                                                                                  |
| created_at          | 2017-01-10T10:48:11+00:00                                                                                                                                                                                                  |
| name                | swarm-cluster_test                                                                                                                                                                                                         |
| updated_at          | 2017-01-10T11:48:19+00:00                                                                                                                                                                                                  |
| discovery_url       | https://discovery.etcd.io/77b79dca8dbece069155914d61f77d9b                                                                                                                                                                 |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)', '0': 'resources[0]: Stack CREATE cancelled'} |
| api_address         | -                                                                                                                                                                                                                          |
| coe_version         | -                                                                                                                                                                                                                          |
| master_addresses    | []                                                                                                                                                                                                                         |
| create_timeout      | 60                                                                                                                                                                                                                         |
| node_addresses      | []                                                                                                                                                                                                                         |
| master_count        | 1                                                                                                                                                                                                                          |
| container_version   | -                                                                                                                                                                                                                          |
| node_count          | 1                                                                                                                                                                                                                          |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next...

 [hpchost1@controller ~]$ heat stack-list -n | grep swarm-cluster_test
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| 89003fb7-a094-4ed8-b744-00121e8cd577 | swarm-cluster_test-nftqb3b77za6                                                                            | CREATE_FAILED   | 2017-01-10T10:48:09Z | None                 | None                                 |
| 10655a90-845e-4b7b-8fce-2f7e7a97ae15 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws                                                 | CREATE_FAILED   | 2017-01-10T10:48:29Z | None                 | 89003fb7-a094-4ed8-b744-00121e8cd577 |
| eeebab48-bfe8-449c-a39f-341d99e88074 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52                                  | CREATE_FAILED   | 2017-01-10T10:48:38Z | None                 | 10655a90-845e-4b7b-8fce-2f7e7a97ae15 |
| dd19704c-9261-4a6c-8ad2-6f667c78697e | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-etcd_address_switch-hzykf3thzdha | CREATE_COMPLETE | 2017-01-10T10:48:44Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |
| 03c01386-71e2-41c6-ad28-6d63c66ee110 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-api_address_switch-lprrlk3rh6p6  | CREATE_COMPLETE | 2017-01-10T10:48:48Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |

next...

[hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | master_wait_condition               |                                      | OS::Heat::WaitCondition                      | CREATE_FAILED   | 2017-01-10T10:48:39Z |

    [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | 0             | eeebab48-bfe8-449c-a39f-341d99e88074 | file:///usr/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2017-01-10T10:48:29Z |

    [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6 | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | swarm_masters       | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                | OS::Heat::ResourceGroup                         | CREATE_FAILED   | 2017-01-10T10:48:11Z |
    [hpchost1@controller ~]$

next...

 [hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
[hpchost1@controller ~]$

[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 swarm_masters
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                      |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                          |
|                        |   "attributes": null,                                                                                                                                                      |
|                        |   "refs": null,                                                                                                                                                            |
|                        |   "refs_map": null,                                                                                                                                                        |
|                        |   "removed_rsrc_list": []                                                                                                                                                  |
|                        | }                                                                                                                                                                          |
| creation_time          | 2017-01-10T10:48:11Z                                                                                                                                                       |
| description            |                                                                                                                                                                            |
| links                  | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6/89003fb7-a094-4ed8-b744-00121e8cd577/resources/swarm_masters (self)      |
|                        | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6/89003fb7-a094-4ed8-b744-00121e8cd577 (stack)                             |
|                        | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws/10655a90-845e-4b7b-8fce-2f7e7a97ae15 (nested) |
| logical_resource_id    | swarm_masters                                                                                                                                                              |
| physical_resource_id   | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                                                                                                       |
| required_by            | etcd_address_switch                                                                                                                                                        |
|                        | api_address_switch                                                                                                                                                         |
| resource_name          | swarm_masters                                                                                                                                                              |
| resource_status        | CREATE_FAILED                                                                                                                                                              |
| resource_status_reason | CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)                   |
| resource_type          | OS::Heat::ResourceGroup                                                                                                                                                    |
| updated_time           | 2017-01-10T10:48:11Z                                                                                                                                                       |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

heat-engine.log file on controller node..

2017-01-10 11:48:09.256 4105 INFO heat.engine.scheduler [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] timed out
2017-01-10 11:48:09.286 4102 INFO heat.engine.service [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Starting cancel of updating stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws
2017-01-10 11:48:09.336 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Stack CREATE cancelled
2017-01-10 11:48:09.383 4105 INFO heat.engine.stack [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Stack CREATE FAILED ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2017-01-10 16:19:22 -0500

zaneb gravatar image

Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received indicates that the agent on the server was unable to signal back to Heat that the server had booted successfully. This could be due to a problem with the image, a problem with the networking between the server and Heat, or some other problem along this path.

edit flag offensive delete link more

Comments

@ Zaneb, Thank you so much for your reply. Just to make sure we are on the same page of understanding, Which agent are you taking about(is it agent inside swarm-cluster master)? I am using fedora-atomic-latest image following RDO centos 7 openstack newton guide.

Mutty Putty gravatar imageMutty Putty ( 2017-01-10 23:07:47 -0500 )edit

Me to think the same, agent is unable to send the signal back to heat that stack successfully created. could you please tell me troubleshooting procedure in details? I have noticed that out of 5 stacks of the cluster, 2 stacks getting successfully completed quickly and after log time rest failed.

Mutty Putty gravatar imageMutty Putty ( 2017-01-10 23:10:45 -0500 )edit
-1

answered 2016-11-25 05:23:17 -0500

ashish235 gravatar image

Same issue.

2016-11-25 16:21:24.062 13858 ERROR heat.engine.resource [req-51007baf-2b3c-45b0-bfad-613a542b12de 16bfd673590e48069db821c9128a7c72 527da6b91b9c49afb97a2c286fa5d2f9 - - -] Resource type OS::Neutron::RouterInterface unavailable 2016-11-25 16:21:24.062 13858 ERROR heat.engine.resource Traceback (most recent call last):

edit flag offensive delete link more

Comments

@ashish, did you get any answer?

Mutty Putty gravatar imageMutty Putty ( 2016-12-05 03:34:07 -0500 )edit

Nopes :| @mutty

ashish235 gravatar imageashish235 ( 2017-01-23 08:22:07 -0500 )edit

Had the same issue...make sure that the DNS server you set up for your template can resolve where your heat process is running (ie. controller if you are following the docs). Also make sure that the Swarm or Kubernetes node has access to talk back to the controller to notify heat (ie.public net)

proceonmw gravatar imageproceonmw ( 2017-01-27 12:31:14 -0500 )edit

@proceonmw didn't work for me. Gave the address of a custom DNS to the template and the network is de default "provider" from the docs

FernaG gravatar imageFernaG ( 2017-02-17 05:03:07 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-11-24 07:39:30 -0500

Seen: 257 times

Last updated: Jan 10