Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Swarm cluster-create fails

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on Ubuntu 16.04. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

hpchost1@controller:~$ magnum cluster-show swarm-cluster
    +---------------------+------------------------------------------------------------+
    | Property            | Value                                                      |
    +---------------------+------------------------------------------------------------+
    | status              | CREATE_IN_PROGRESS                                         |
    | cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0                       |
    | node_addresses      | []                                                         |
    | uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8                       |
    | stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6                       |
    | status_reason       | -                                                          |
    | created_at          | 2016-11-24T10:17:03+00:00                                  |
    | updated_at          | -                                                          |
    | coe_version         | -                                                          |
    | keypair             | Containerkey                                               |
    | api_address         | -                                                          |
    | master_addresses    | []                                                         |
    | create_timeout      | 60                                                         |
    | node_count          | 1                                                          |
    | discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63 |
    | master_count        | 1                                                          |
    | container_version   | -                                                          |
    | name                | swarm-cluster                                              |
    +---------------------+------------------------------------------------------------+


hpchost1@controller:~$ heat stack-list -n | grep swarm-cluster
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_IN_PROGRESS | 2016-11-24T10:17:01Z | None         | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_IN_PROGRESS | 2016-11-24T10:17:09Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_IN_PROGRESS | 2016-11-24T10:17:11Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 | CREATE_COMPLETE    | 2016-11-24T10:17:17Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23  | CREATE_COMPLETE    | 2016-11-24T10:17:19Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |

next......

hpchost1@controller:~$ magnum cluster-show swarm-cluster
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                                                                                                                    |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                                                                                                                            |
| cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0                                                                                                                                                                                                                                                                                     |
| node_addresses      | []                                                                                                                                                                                                                                                                                                                       |
| uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8                                                                                                                                                                                                                                                                                     |
| stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6                                                                                                                                                                                                                                                                                     |
| status_reason       | Timed out                                                                                                                                                                                                                                                                                                                |
| created_at          | 2016-11-24T10:17:03+00:00                                                                                                                                                                                                                                                                                                |
| updated_at          | 2016-11-24T11:17:25+00:00                                                                                                                                                                                                                                                                                                |
| coe_version         | -                                                                                                                                                                                                                                                                                                                        |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] Timed out)', 'master_wait_condition': 'WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received', '0': 'resources[0]: Stack CREATE cancelled'} |
| keypair             | Containerkey                                                                                                                                                                                                                                                                                                             |
| api_address         | -                                                                                                                                                                                                                                                                                                                        |
| master_addresses    | []                                                                                                                                                                                                                                                                                                                       |
| create_timeout      | 60                                                                                                                                                                                                                                                                                                                       |
| node_count          | 1                                                                                                                                                                                                                                                                                                                        |
| discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63                                                                                                                                                                                                                                                               |
| master_count        | 1                                                                                                                                                                                                                                                                                                                        |
| container_version   | -                                                                                                                                                                                                                                                                                                                        |
| name                | swarm-cluster                                                                                                                                                                                                                                                                                                            |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next....

hpchost1@controller:~$ heat stack-list -n | grep swarm-cluster
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_FAILED   | 2016-11-24T10:17:01Z | None         | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_FAILED   | 2016-11-24T10:17:09Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_FAILED   | 2016-11-24T10:17:11Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 | CREATE_COMPLETE | 2016-11-24T10:17:17Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23  | CREATE_COMPLETE | 2016-11-24T10:17:19Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |

next...

hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| swarm_masters       | ba7e8398-8966-4632-a08b-02d52c7c71b3

next...

hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| 0             | f19adfeb-2a0a-454a-8c45-0b5432a0c78f | file:///var/lib/magnum/env/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2016-11-24T10:17:09Z |

next...

 hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | master_wait_condition               |                                      | OS::Heat::WaitCondition                      | CREATE_FAILED   | 2016-11-24T10:17:11Z |

next...

hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition

heat-engine.log file on controller node..

2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::Router unavailable
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
                                                                                                                                                                                           1050,1        83%
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a2f85415-021d-4de7-ac6d-af9bc24413a6']
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8e104664-7720-4b6e-bb83-3b745f499dd4']
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-44186708-ad7a-4a75-b6ba-e754f700dbf5']
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
                                                                                                                                                                                           1143,1        91%
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-9bdaadec-c9d8-42c0-bd7a-7bbef01a40fe']
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
                                                                                                                                                                                           1197,1        95%
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a932d582-0555-4312-ab52-5046ce97b70f']
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource
2016-11-24 16:47:01.407 2907 INFO heat.engine.scheduler [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] timed out
2016-11-24 16:47:01.432 2905 INFO heat.engine.service [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Starting cancel of updating stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf
2016-11-24 16:47:01.555 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Stack CREATE cancelled
2016-11-24 16:47:01.647 2907 INFO heat.engine.stack [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f): Timed out
2016-11-24 16:47:01.716 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 INFO heat.engine.resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] CREATE: TemplateResource "0" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf" [ba7e8398-8966-4632-a08b-02d52c7c71b3]
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     yield
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 354, in check_create_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     return self._check_status_complete(self.CREATE)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 404, in _check_status_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     action=action)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource
2016-11-24 16:47:02.300 2900 INFO heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.310 2900 WARNING heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf with traversal ID a4846681-4fc6-4ed0-a451-9b0a20cf1511, to CREATE_FAILED
2016-11-24 16:47:02.325 2900 INFO heat.engine.check_resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Re-trigger resource: (212, True)
2016-11-24 17:27:49.617 2908 INFO heat.engine.resources.openstack.heat.wait_condition [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Timed out (0 of 1 received)
2016-11-24 17:27:49.618 2908 INFO heat.engine.resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f]
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     return self._wait(*data)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource
2016-11-24 17:27:49.758 2908 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2016-11-24 17:27:49.769 2908 WARNING heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc with traversal ID b24d829c-7117-42a4-9bc8-44a8874bbf8b, to CREATE_FAILED
2016-11-24 17:27:49.787 2908 INFO heat.engine.check_resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Re-trigger resource: (215, True)

I am unable to find out what’s wrong.

Thanks in advance.

Any help will be appreciated.

Swarm cluster-create fails

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on Ubuntu 16.04. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

hpchost1@controller:~$ magnum cluster-show swarm-cluster
    +---------------------+------------------------------------------------------------+
    | Property            | Value                                                      |
    +---------------------+------------------------------------------------------------+
    | status              | CREATE_IN_PROGRESS                                         |
    | cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0                       |
    | node_addresses      | []                                                         |
    | uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8                       |
    | stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6                       |
    | status_reason       | -                                                          |
    | created_at          | 2016-11-24T10:17:03+00:00                                  |
    | updated_at          | -                                                          |
    | coe_version         | -                                                          |
    | keypair             | Containerkey                                               |
    | api_address         | -                                                          |
    | master_addresses    | []                                                         |
    | create_timeout      | 60                                                         |
    | node_count          | 1                                                          |
    | discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63 |
    | master_count        | 1                                                          |
    | container_version   | -                                                          |
    | name                | swarm-cluster                                              |
    +---------------------+------------------------------------------------------------+


hpchost1@controller:~$ heat stack-list -n | grep swarm-cluster
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_IN_PROGRESS | 2016-11-24T10:17:01Z | None         | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_IN_PROGRESS | 2016-11-24T10:17:09Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_IN_PROGRESS | 2016-11-24T10:17:11Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 | CREATE_COMPLETE    | 2016-11-24T10:17:17Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23  | CREATE_COMPLETE    | 2016-11-24T10:17:19Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |

next......

hpchost1@controller:~$ magnum cluster-show swarm-cluster
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                                                                                                                    |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                                                                                                                            |
| cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0                                                                                                                                                                                                                                                                                     |
| node_addresses      | []                                                                                                                                                                                                                                                                                                                       |
| uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8                                                                                                                                                                                                                                                                                     |
| stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6                                                                                                                                                                                                                                                                                     |
| status_reason       | Timed out                                                                                                                                                                                                                                                                                                                |
| created_at          | 2016-11-24T10:17:03+00:00                                                                                                                                                                                                                                                                                                |
| updated_at          | 2016-11-24T11:17:25+00:00                                                                                                                                                                                                                                                                                                |
| coe_version         | -                                                                                                                                                                                                                                                                                                                        |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] Timed out)', 'master_wait_condition': 'WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received', '0': 'resources[0]: Stack CREATE cancelled'} |
| keypair             | Containerkey                                                                                                                                                                                                                                                                                                             |
| api_address         | -                                                                                                                                                                                                                                                                                                                        |
| master_addresses    | []                                                                                                                                                                                                                                                                                                                       |
| create_timeout      | 60                                                                                                                                                                                                                                                                                                                       |
| node_count          | 1                                                                                                                                                                                                                                                                                                                        |
| discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63                                                                                                                                                                                                                                                               |
| master_count        | 1                                                                                                                                                                                                                                                                                                                        |
| container_version   | -                                                                                                                                                                                                                                                                                                                        |
| name                | swarm-cluster                                                                                                                                                                                                                                                                                                            |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next....

hpchost1@controller:~$ heat stack-list -n | grep swarm-cluster
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_FAILED   | 2016-11-24T10:17:01Z | None         | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_FAILED   | 2016-11-24T10:17:09Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_FAILED   | 2016-11-24T10:17:11Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 | CREATE_COMPLETE | 2016-11-24T10:17:17Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23  | CREATE_COMPLETE | 2016-11-24T10:17:19Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |

next...

hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| swarm_masters       | ba7e8398-8966-4632-a08b-02d52c7c71b3

next...

hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| 0             | f19adfeb-2a0a-454a-8c45-0b5432a0c78f | file:///var/lib/magnum/env/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2016-11-24T10:17:09Z |

next...

 hpchost1@controller:~$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | master_wait_condition               |                                      | OS::Heat::WaitCondition                      | CREATE_FAILED   | 2016-11-24T10:17:11Z |

next...

hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition

heat-engine.log file on controller node..

2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::Router unavailable
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
                                                                                                                                                                                           1050,1        83%
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a2f85415-021d-4de7-ac6d-af9bc24413a6']
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8e104664-7720-4b6e-bb83-3b745f499dd4']
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-44186708-ad7a-4a75-b6ba-e754f700dbf5']
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
                                                                                                                                                                                           1143,1        91%
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-9bdaadec-c9d8-42c0-bd7a-7bbef01a40fe']
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
                                                                                                                                                                                           1197,1        95%
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a932d582-0555-4312-ab52-5046ce97b70f']
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource
2016-11-24 16:47:01.407 2907 INFO heat.engine.scheduler [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] timed out
2016-11-24 16:47:01.432 2905 INFO heat.engine.service [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Starting cancel of updating stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf
2016-11-24 16:47:01.555 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Stack CREATE cancelled
2016-11-24 16:47:01.647 2907 INFO heat.engine.stack [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f): Timed out
2016-11-24 16:47:01.716 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 INFO heat.engine.resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] CREATE: TemplateResource "0" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf" [ba7e8398-8966-4632-a08b-02d52c7c71b3]
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     yield
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 354, in check_create_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     return self._check_status_complete(self.CREATE)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 404, in _check_status_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     action=action)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource
2016-11-24 16:47:02.300 2900 INFO heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.310 2900 WARNING heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf with traversal ID a4846681-4fc6-4ed0-a451-9b0a20cf1511, to CREATE_FAILED
2016-11-24 16:47:02.325 2900 INFO heat.engine.check_resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Re-trigger resource: (212, True)
2016-11-24 17:27:49.617 2908 INFO heat.engine.resources.openstack.heat.wait_condition [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Timed out (0 of 1 received)
2016-11-24 17:27:49.618 2908 INFO heat.engine.resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f]
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     return self._wait(*data)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource
2016-11-24 17:27:49.758 2908 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2016-11-24 17:27:49.769 2908 WARNING heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc with traversal ID b24d829c-7117-42a4-9bc8-44a8874bbf8b, to CREATE_FAILED
2016-11-24 17:27:49.787 2908 INFO heat.engine.check_resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Re-trigger resource: (215, True)

I am unable to find out what’s wrong.

Thanks in advance.

Any help will be appreciated.

Swarm cluster-create fails

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on Ubuntu 16.04. CentOS 7. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

hpchost1@controller:~$  [hpchost1@controller ~]$ magnum cluster-show swarm-cluster
    +---------------------+------------------------------------------------------------+
    swarm-cluster_test
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                       |
    +---------------------+------------------------------------------------------------+
    +---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_IN_PROGRESS CREATE_FAILED                                                                                                                                                                                                              |
 | cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0 15bdcb55-a334-4699-ad86-f98ffa338589                                                                                                                                                                                       |
 | uuid                | 3e6c1f12-e9e9-4fbd-8262-d32d89aa7c3e                                                                                                                                                                                       |
| stack_id            | 89003fb7-a094-4ed8-b744-00121e8cd577                                                                                                                                                                                       |
| status_reason       | Timed out                                                                                                                                                                                                                  |
| created_at          | 2017-01-10T10:48:11+00:00                                                                                                                                                                                                  |
| name                | swarm-cluster_test                                                                                                                                                                                                         |
| updated_at          | 2017-01-10T11:48:19+00:00                                                                                                                                                                                                  |
| discovery_url       | https://discovery.etcd.io/77b79dca8dbece069155914d61f77d9b                                                                                                                                                                 |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)', '0': 'resources[0]: Stack CREATE cancelled'} |
| api_address         | -                                                                                                                                                                                                                          |
| coe_version         | -                                                                                                                                                                                                                          |
| master_addresses    | []                                                                                                                                                                                                                         |
| create_timeout      | 60                                                                                                                                                                                                                         |
| node_addresses      | []                                                          |
    | uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8 | master_count        | 1                                                                                                                                                                                                                          |
    | stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | container_version   | -                                                                                                                                                                                                                          |
    | status_reason       | - | node_count          | 1                                                                                                                                                                                                                          |
    | created_at          | 2016-11-24T10:17:03+00:00                                  |
    | updated_at          | -                                                          |
    | coe_version         | -                                                          |
    | keypair             | Containerkey                                               |
    | api_address         | -                                                          |
    | master_addresses    | []                                                         |
    | create_timeout      | 60                                                         |
    | node_count          | 1                                                          |
    | discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63 |
    | master_count        | 1                                                          |
    | container_version   | -                                                          |
    | name                | swarm-cluster                                              |
    +---------------------+------------------------------------------------------------+
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 hpchost1@controller:~$ 

 [hpchost1@controller ~]$ heat stack-list -n | grep swarm-cluster
swarm-cluster_test
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_IN_PROGRESS | 2016-11-24T10:17:01Z 89003fb7-a094-4ed8-b744-00121e8cd577 | swarm-cluster_test-nftqb3b77za6                                                                            | CREATE_FAILED   | 2017-01-10T10:48:09Z | None          | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_IN_PROGRESS | 2016-11-24T10:17:09Z 10655a90-845e-4b7b-8fce-2f7e7a97ae15 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws                                                 | CREATE_FAILED   | 2017-01-10T10:48:29Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6         | 89003fb7-a094-4ed8-b744-00121e8cd577 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_IN_PROGRESS | 2016-11-24T10:17:11Z eeebab48-bfe8-449c-a39f-341d99e88074 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52                                  | CREATE_FAILED   | 2017-01-10T10:48:38Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3         | 10655a90-845e-4b7b-8fce-2f7e7a97ae15 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 dd19704c-9261-4a6c-8ad2-6f667c78697e | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-etcd_address_switch-hzykf3thzdha | CREATE_COMPLETE    | 2016-11-24T10:17:17Z | 2017-01-10T10:48:44Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f         | eeebab48-bfe8-449c-a39f-341d99e88074 |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23 03c01386-71e2-41c6-ad28-6d63c66ee110 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-api_address_switch-lprrlk3rh6p6  | CREATE_COMPLETE    | 2016-11-24T10:17:19Z | 2017-01-10T10:48:48Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |

next......

hpchost1@controller:~$ magnum cluster-show swarm-cluster
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                                                                                                                    |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                                                                                                                            |
| cluster_template_id | 25d79986-faab-49b2-be8f-3fe547850ce0                                                                                                                                                                                                                                                                                     |
| node_addresses      | []                                                                                                                                                                                                                                                                                                                       |
| uuid                | f52f3acf-7ef9-487b-a38f-a4a4d1ba40d8                                                                                                                                                                                                                                                                                     |
| stack_id            | d5228b3c-32ca-4896-a202-b2f1fb12f6a6                                                                                                                                                                                                                                                                                     |
| status_reason       | Timed out                                                                                                                                                                                                                                                                                                                |
| created_at          | 2016-11-24T10:17:03+00:00                                                                                                                                                                                                                                                                                                |
| updated_at          | 2016-11-24T11:17:25+00:00                                                                                                                                                                                                                                                                                                |
| coe_version         | -                                                                                                                                                                                                                                                                                                                        |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] Timed out)', 'master_wait_condition': 'WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received', '0': 'resources[0]: Stack CREATE cancelled'} |
| keypair             | Containerkey                                                                                                                                                                                                                                                                                                             |
| api_address         | -                                                                                                                                                                                                                                                                                                                        |
| master_addresses    | []                                                                                                                                                                                                                                                                                                                       |
| create_timeout      | 60                                                                                                                                                                                                                                                                                                                       |
| node_count          | 1                                                                                                                                                                                                                                                                                                                        |
| discovery_url       | https://discovery.etcd.io/abe196231709089d5d369ddc79de3b63                                                                                                                                                                                                                                                               |
| master_count        | 1                                                                                                                                                                                                                                                                                                                        |
| container_version   | -                                                                                                                                                                                                                                                                                                                        |
| name                | swarm-cluster                                                                                                                                                                                                                                                                                                            |
+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next....

hpchost1@controller:~$ heat stack-list -n | grep swarm-cluster
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| d5228b3c-32ca-4896-a202-b2f1fb12f6a6 | swarm-cluster-ot7r7jhas24f                                                                            | CREATE_FAILED   | 2016-11-24T10:17:01Z | None         | None                                 |
| ba7e8398-8966-4632-a08b-02d52c7c71b3 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf                                                 | CREATE_FAILED   | 2016-11-24T10:17:09Z | None         | d5228b3c-32ca-4896-a202-b2f1fb12f6a6 |
| f19adfeb-2a0a-454a-8c45-0b5432a0c78f | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc                                  | CREATE_FAILED   | 2016-11-24T10:17:11Z | None         | ba7e8398-8966-4632-a08b-02d52c7c71b3 |
| 2b011d1c-dc11-48ee-90dc-1a3640f08c73 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-etcd_address_switch-4ozjmkcsfat7 | CREATE_COMPLETE | 2016-11-24T10:17:17Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f |
| 0877d5de-37da-4d9e-8614-11604939fd66 | swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc-api_address_switch-h3objfpyjk23  | CREATE_COMPLETE | 2016-11-24T10:17:19Z | None         | f19adfeb-2a0a-454a-8c45-0b5432a0c78f         | eeebab48-bfe8-449c-a39f-341d99e88074 |

next...

hpchost1@controller:~$ [hpchost1@controller ~]$ heat resource-list swarm-cluster-ot7r7jhas24f swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| swarm_masters       | ba7e8398-8966-4632-a08b-02d52c7c71b3
master_wait_condition | | OS::Heat::WaitCondition | CREATE_FAILED | 2017-01-10T10:48:39Z |

next...

hpchost1@controller:~$ [hpchost1@controller ~]$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws | grep "FAILED"
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
| 0             | f19adfeb-2a0a-454a-8c45-0b5432a0c78f | file:///var/lib/magnum/env/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml eeebab48-bfe8-449c-a39f-341d99e88074 | file:///usr/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2016-11-24T10:17:09Z 2017-01-10T10:48:29Z |

next...

 hpchost1@controller:~$ [hpchost1@controller ~]$ heat resource-list swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc swarm-cluster_test-nftqb3b77za6 | grep "FAILED"
 WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | master_wait_condition               |                                      | OS::Heat::WaitCondition | swarm_masters       | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                | OS::Heat::ResourceGroup                         | CREATE_FAILED   | 2016-11-24T10:17:11Z 2017-01-10T10:48:11Z |
[hpchost1@controller ~]$ 


next...

hpchost1@controller:~$ 

 [hpchost1@controller ~]$ heat resource-show swarm-cluster-ot7r7jhas24f swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf OS::Heat::ResourceGroup
hpchost1@controller:~$ heat resource-show swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc OS::Heat::WaitCondition
[hpchost1@controller ~]$

heat-engine.log file on controller node..

2016-11-24 16:46:55.017 2908 ERROR 2017-01-10 11:48:09.256 4105 INFO heat.engine.scheduler [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] timed out
2017-01-10 11:48:09.286 4102 INFO heat.engine.service [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Starting cancel of updating stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws
2017-01-10 11:48:09.336 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Stack CREATE cancelled
2017-01-10 11:48:09.383 4105 INFO heat.engine.stack [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6): Timed out
2017-01-10 11:48:09.405 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 INFO heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Resource type OS::Neutron::Router unavailable
2016-11-24 16:46:55.017 2908 CREATE: TemplateResource "0" [eeebab48-bfe8-449c-a39f-341d99e88074] Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws" [10655a90-845e-4b7b-8fce-2f7e7a97ae15]
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.017 2908 2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 184, 753, in _validate_service_availability
2016-11-24 16:46:55.017 2908 _action_recorder
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.017 2908 yield
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 699, 855, in is_service_available
2016-11-24 16:46:55.017 2908 _do_action
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.017 2908 yield self.action_handler_task(action, args=handler_args)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 87, 353, in has_extension
2016-11-24 16:46:55.017 2908 wrapper
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.017 2908 step = next(subtask)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1220, 806, in decorate
2016-11-24 16:46:55.017 2908 action_handler_task
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.017 2908 done = check(handler_data)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 825, 355, in get_or_create
2016-11-24 16:46:55.017 2908 check_create_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.017 2908 return self._check_status_complete(self.CREATE)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 154, 405, in __enter__
2016-11-24 16:46:55.017 2908 _check_status_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.017 2908 action=action)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
                                                                                                                                                                                           1050,1        83%
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.017 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a2f85415-021d-4de7-ac6d-af9bc24413a6']
2016-11-24 16:46:55.017 2908 ResourceFailure: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource
2016-11-24 16:46:55.044 2908 ERROR 2017-01-10 11:48:10.479 4108 INFO heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.488 4108 WARNING heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Failed to set state of stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws with traversal ID 21567a10-300d-4119-b1a1-f140792a6b0c, to CREATE_FAILED
2017-01-10 11:48:10.510 4108 INFO heat.engine.check_resource [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Re-trigger resource: (68, True)
2017-01-10 12:17:48.436 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:48.457 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:53.246 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:17:53.256 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:29:31.935 4085 INFO heat.engine.resources.openstack.heat.wait_condition [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074] Timed out (0 of 1 received)
2017-01-10 12:29:31.936 4085 INFO heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.044 2908 CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074]
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.044 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8e104664-7720-4b6e-bb83-3b745f499dd4']
2016-11-24 16:46:55.044 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-44186708-ad7a-4a75-b6ba-e754f700dbf5']
2016-11-24 16:46:55.058 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
                                                                                                                                                                                           1143,1        91%
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-9bdaadec-c9d8-42c0-bd7a-7bbef01a40fe']
2016-11-24 16:46:55.071 2908 ERROR heat.engine.resource
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource [req-22e3fa0f-2507-4f3c-b64e-d052ef8d1b32 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Resource type OS::Neutron::RouterInterface unavailable
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 184, in _validate_service_availability
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     (svc_available, reason) = cls.is_service_available(context)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 699, in is_service_available
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     req_extension))
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 87, in has_extension
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return alias in self._list_extensions()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1220, in decorate
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     should_cache_fn)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 825, in get_or_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     async_creator) as value:
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self._enter()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 94, in _enter
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     generated = self._enter_create(createdtime)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/lock.py", line 145, in _enter_create
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created = self.creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 792, in gen_value
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     created_value = creator()
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1216, in creator
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return fn(*arg, **kw)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/neutron/__init__.py", line 82, in _list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     extensions = self.client().list_extensions().get('extensions')
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 732, in list_extensions
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     return self.get(self.extensions_path, params=_params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 360, in get
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 337, in retry_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     headers=headers, params=params)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
                                                                                                                                                                                           1197,1        95%
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource     request_ids=request_ids)
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource Neutron server returns request_ids: ['req-a932d582-0555-4312-ab52-5046ce97b70f']
2016-11-24 16:46:55.086 2908 ERROR heat.engine.resource
2016-11-24 16:47:01.407 2907 INFO heat.engine.scheduler [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster-ot7r7jhas24f" [d5228b3c-32ca-4896-a202-b2f1fb12f6a6] timed out
2016-11-24 16:47:01.432 2905 INFO heat.engine.service [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Starting cancel of updating stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf
2016-11-24 16:47:01.555 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Stack CREATE cancelled
2016-11-24 16:47:01.647 2907 INFO heat.engine.stack [req-974e3b0a-3117-4738-995e-5487cc402333 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f): Timed out
2016-11-24 16:47:01.716 2905 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 INFO heat.engine.resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] CREATE: TemplateResource "0" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf" [ba7e8398-8966-4632-a08b-02d52c7c71b3]
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 16:47:02.140 2900 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 354, in check_create_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     return self._check_status_complete(self.CREATE)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/stack_resource.py", line 404, in _check_status_complete
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource     action=action)
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.140 2900 ERROR heat.engine.resource
2016-11-24 16:47:02.300 2900 INFO heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2016-11-24 16:47:02.310 2900 WARNING heat.engine.stack [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf with traversal ID a4846681-4fc6-4ed0-a451-9b0a20cf1511, to CREATE_FAILED
2016-11-24 16:47:02.325 2900 INFO heat.engine.check_resource [req-60dd43d4-c9ce-4d65-9e60-d2f5b565e411 - - - - -] Re-trigger resource: (212, True)
2016-11-24 17:27:49.617 2908 INFO heat.engine.resources.openstack.heat.wait_condition [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f] Timed out (0 of 1 received)
2016-11-24 17:27:49.618 2908 INFO heat.engine.resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 - - -] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc" [f19adfeb-2a0a-454a-8c45-0b5432a0c78f]
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource Traceback (most recent call last):
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 753, in _action_recorder
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 855, in _do_action
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 353, in wrapper
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     step = next(subtask)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 806, in action_handler_task
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     done = check(handler_data)
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2016-11-24 17:27:49.618 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     return self._wait(*data)
2016-11-24 17:27:49.618 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/heat/wait_condition.py", "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2016-11-24 17:27:49.618 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource     raise exc
2016-11-24 17:27:49.618 2908 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2016-11-24 17:27:49.618 2908 2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource
2016-11-24 17:27:49.758 2908 2017-01-10 12:29:31.999 4085 INFO heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Stack CREATE FAILED (swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc): (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2016-11-24 17:27:49.769 2908 2017-01-10 12:29:32.014 4085 WARNING heat.engine.stack [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Failed to set state of stack swarm-cluster-ot7r7jhas24f-swarm_masters-u3cc7s6f3dhf-0-eckygeivkwnc swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 with traversal ID b24d829c-7117-42a4-9bc8-44a8874bbf8b, e5ef9bd4-8eab-4bec-bd42-d58b1a66828f, to CREATE_FAILED
2016-11-24 17:27:49.787 2908 2017-01-10 12:29:32.039 4085 INFO heat.engine.check_resource [req-d88a783b-509b-446e-bc6e-cae63c299b1f 2793f25d51c949f4bc18311b7cd1a380 b5a63ca9515840b58284e09871b5c188 [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Re-trigger resource: (215, (71, True)

I am unable to find out what’s wrong.

Thanks in advance.

Any help will be appreciated. appreciated.

Swarm cluster-create fails

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on CentOS 7. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

 [hpchost1@controller ~]$ magnum cluster-show swarm-cluster_test
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                      |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                              |
| cluster_template_id | 15bdcb55-a334-4699-ad86-f98ffa338589                                                                                                                                                                                       |
| uuid                | 3e6c1f12-e9e9-4fbd-8262-d32d89aa7c3e                                                                                                                                                                                       |
| stack_id            | 89003fb7-a094-4ed8-b744-00121e8cd577                                                                                                                                                                                       |
| status_reason       | Timed out                                                                                                                                                                                                                  |
| created_at          | 2017-01-10T10:48:11+00:00                                                                                                                                                                                                  |
| name                | swarm-cluster_test                                                                                                                                                                                                         |
| updated_at          | 2017-01-10T11:48:19+00:00                                                                                                                                                                                                  |
| discovery_url       | https://discovery.etcd.io/77b79dca8dbece069155914d61f77d9b                                                                                                                                                                 |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)', '0': 'resources[0]: Stack CREATE cancelled'} |
| api_address         | -                                                                                                                                                                                                                          |
| coe_version         | -                                                                                                                                                                                                                          |
| master_addresses    | []                                                                                                                                                                                                                         |
| create_timeout      | 60                                                                                                                                                                                                                         |
| node_addresses      | []                                                                                                                                                                                                                         |
| master_count        | 1                                                                                                                                                                                                                          |
| container_version   | -                                                                                                                                                                                                                          |
| node_count          | 1                                                                                                                                                                                                                          |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next...

 [hpchost1@controller ~]$ heat stack-list -n | grep swarm-cluster_test
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| 89003fb7-a094-4ed8-b744-00121e8cd577 | swarm-cluster_test-nftqb3b77za6                                                                            | CREATE_FAILED   | 2017-01-10T10:48:09Z | None                 | None                                 |
| 10655a90-845e-4b7b-8fce-2f7e7a97ae15 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws                                                 | CREATE_FAILED   | 2017-01-10T10:48:29Z | None                 | 89003fb7-a094-4ed8-b744-00121e8cd577 |
| eeebab48-bfe8-449c-a39f-341d99e88074 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52                                  | CREATE_FAILED   | 2017-01-10T10:48:38Z | None                 | 10655a90-845e-4b7b-8fce-2f7e7a97ae15 |
| dd19704c-9261-4a6c-8ad2-6f667c78697e | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-etcd_address_switch-hzykf3thzdha | CREATE_COMPLETE | 2017-01-10T10:48:44Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |
| 03c01386-71e2-41c6-ad28-6d63c66ee110 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-api_address_switch-lprrlk3rh6p6  | CREATE_COMPLETE | 2017-01-10T10:48:48Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |

next...

[hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 | grep "FAILED"
 WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
 | master_wait_condition               |                                      | OS::Heat::WaitCondition                      | CREATE_FAILED   | 2017-01-10T10:48:39Z |

 [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws | grep "FAILED"
 WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
 | 0             | eeebab48-bfe8-449c-a39f-341d99e88074 | file:///usr/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2017-01-10T10:48:29Z |

 [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6 | grep "FAILED"
 WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
 | swarm_masters       | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                | OS::Heat::ResourceGroup                         | CREATE_FAILED   | 2017-01-10T10:48:11Z |
 [hpchost1@controller ~]$ ~]$
next...

next...

 [hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
[hpchost1@controller ~]$

heat-engine.log file on controller node..

2017-01-10 11:48:09.256 4105 INFO heat.engine.scheduler [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] timed out
2017-01-10 11:48:09.286 4102 INFO heat.engine.service [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Starting cancel of updating stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws
2017-01-10 11:48:09.336 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Stack CREATE cancelled
2017-01-10 11:48:09.383 4105 INFO heat.engine.stack [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6): Timed out
2017-01-10 11:48:09.405 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 INFO heat.engine.resource [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] CREATE: TemplateResource "0" [eeebab48-bfe8-449c-a39f-341d99e88074] Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws" [10655a90-845e-4b7b-8fce-2f7e7a97ae15]
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource Traceback (most recent call last):
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     yield
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 855, in _do_action
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     step = next(subtask)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 806, in action_handler_task
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     done = check(handler_data)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 355, in check_create_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     return self._check_status_complete(self.CREATE)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 405, in _check_status_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     action=action)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource
2017-01-10 11:48:10.479 4108 INFO heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.488 4108 WARNING heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Failed to set state of stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws with traversal ID 21567a10-300d-4119-b1a1-f140792a6b0c, to CREATE_FAILED
2017-01-10 11:48:10.510 4108 INFO heat.engine.check_resource [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Re-trigger resource: (68, True)
2017-01-10 12:17:48.436 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:48.457 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:53.246 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:17:53.256 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:29:31.935 4085 INFO heat.engine.resources.openstack.heat.wait_condition [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074] Timed out (0 of 1 received)
2017-01-10 12:29:31.936 4085 INFO heat.engine.resource [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074]
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource Traceback (most recent call last):
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 855, in _do_action
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     step = next(subtask)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 806, in action_handler_task
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     done = check(handler_data)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     return self._wait(*data)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     raise exc
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource
2017-01-10 12:29:31.999 4085 INFO heat.engine.stack [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2017-01-10 12:29:32.014 4085 WARNING heat.engine.stack [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Failed to set state of stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 with traversal ID e5ef9bd4-8eab-4bec-bd42-d58b1a66828f, to CREATE_FAILED
2017-01-10 12:29:32.039 4085 INFO heat.engine.check_resource [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Re-trigger resource: (71, True)

I am unable to find out what’s wrong.

Thanks in advance.

Any help will be appreciated.

Swarm cluster-create fails

Hi Everyone,

             I have installed and configured the Container Infrastructure Management service, code-named magnum of Newton OpenStack on CentOS 7. But I am unable to create  Docker Swarm cluster.

Here is my all observation.

 [hpchost1@controller ~]$ magnum cluster-show swarm-cluster_test
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property            | Value                                                                                                                                                                                                                      |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status              | CREATE_FAILED                                                                                                                                                                                                              |
| cluster_template_id | 15bdcb55-a334-4699-ad86-f98ffa338589                                                                                                                                                                                       |
| uuid                | 3e6c1f12-e9e9-4fbd-8262-d32d89aa7c3e                                                                                                                                                                                       |
| stack_id            | 89003fb7-a094-4ed8-b744-00121e8cd577                                                                                                                                                                                       |
| status_reason       | Timed out                                                                                                                                                                                                                  |
| created_at          | 2017-01-10T10:48:11+00:00                                                                                                                                                                                                  |
| name                | swarm-cluster_test                                                                                                                                                                                                         |
| updated_at          | 2017-01-10T11:48:19+00:00                                                                                                                                                                                                  |
| discovery_url       | https://discovery.etcd.io/77b79dca8dbece069155914d61f77d9b                                                                                                                                                                 |
| faults              | {'swarm_masters': 'CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)', '0': 'resources[0]: Stack CREATE cancelled'} |
| api_address         | -                                                                                                                                                                                                                          |
| coe_version         | -                                                                                                                                                                                                                          |
| master_addresses    | []                                                                                                                                                                                                                         |
| create_timeout      | 60                                                                                                                                                                                                                         |
| node_addresses      | []                                                                                                                                                                                                                         |
| master_count        | 1                                                                                                                                                                                                                          |
| container_version   | -                                                                                                                                                                                                                          |
| node_count          | 1                                                                                                                                                                                                                          |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

next...

 [hpchost1@controller ~]$ heat stack-list -n | grep swarm-cluster_test
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
| 89003fb7-a094-4ed8-b744-00121e8cd577 | swarm-cluster_test-nftqb3b77za6                                                                            | CREATE_FAILED   | 2017-01-10T10:48:09Z | None                 | None                                 |
| 10655a90-845e-4b7b-8fce-2f7e7a97ae15 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws                                                 | CREATE_FAILED   | 2017-01-10T10:48:29Z | None                 | 89003fb7-a094-4ed8-b744-00121e8cd577 |
| eeebab48-bfe8-449c-a39f-341d99e88074 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52                                  | CREATE_FAILED   | 2017-01-10T10:48:38Z | None                 | 10655a90-845e-4b7b-8fce-2f7e7a97ae15 |
| dd19704c-9261-4a6c-8ad2-6f667c78697e | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-etcd_address_switch-hzykf3thzdha | CREATE_COMPLETE | 2017-01-10T10:48:44Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |
| 03c01386-71e2-41c6-ad28-6d63c66ee110 | swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52-api_address_switch-lprrlk3rh6p6  | CREATE_COMPLETE | 2017-01-10T10:48:48Z | None                 | eeebab48-bfe8-449c-a39f-341d99e88074 |

next...

[hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | master_wait_condition               |                                      | OS::Heat::WaitCondition                      | CREATE_FAILED   | 2017-01-10T10:48:39Z |

    [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | 0             | eeebab48-bfe8-449c-a39f-341d99e88074 | file:///usr/lib/python2.7/site-packages/magnum/drivers/swarm_fedora_atomic_v1/templates/swarmmaster.yaml | CREATE_FAILED   | 2017-01-10T10:48:29Z |

    [hpchost1@controller ~]$ heat resource-list swarm-cluster_test-nftqb3b77za6 | grep "FAILED"
    WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
    | swarm_masters       | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                | OS::Heat::ResourceGroup                         | CREATE_FAILED   | 2017-01-10T10:48:11Z |
    [hpchost1@controller ~]$

next...

 [hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 OS::Heat::WaitCondition
[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
Stack or resource not found: swarm-cluster_test-nftqb3b77za6 OS::Heat::ResourceGroup
[hpchost1@controller ~]$

[hpchost1@controller ~]$ heat resource-show swarm-cluster_test-nftqb3b77za6 swarm_masters
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                      |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {                                                                                                                                                                          |
|                        |   "attributes": null,                                                                                                                                                      |
|                        |   "refs": null,                                                                                                                                                            |
|                        |   "refs_map": null,                                                                                                                                                        |
|                        |   "removed_rsrc_list": []                                                                                                                                                  |
|                        | }                                                                                                                                                                          |
| creation_time          | 2017-01-10T10:48:11Z                                                                                                                                                       |
| description            |                                                                                                                                                                            |
| links                  | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6/89003fb7-a094-4ed8-b744-00121e8cd577/resources/swarm_masters (self)      |
|                        | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6/89003fb7-a094-4ed8-b744-00121e8cd577 (stack)                             |
|                        | http://controller:8004/v1/282838c02c784f7ab8c89dd37ccfa87c/stacks/swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws/10655a90-845e-4b7b-8fce-2f7e7a97ae15 (nested) |
| logical_resource_id    | swarm_masters                                                                                                                                                              |
| physical_resource_id   | 10655a90-845e-4b7b-8fce-2f7e7a97ae15                                                                                                                                       |
| required_by            | etcd_address_switch                                                                                                                                                        |
|                        | api_address_switch                                                                                                                                                         |
| resource_name          | swarm_masters                                                                                                                                                              |
| resource_status        | CREATE_FAILED                                                                                                                                                              |
| resource_status_reason | CREATE aborted (Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] Timed out)                   |
| resource_type          | OS::Heat::ResourceGroup                                                                                                                                                    |
| updated_time           | 2017-01-10T10:48:11Z                                                                                                                                                       |
+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

heat-engine.log file on controller node..

2017-01-10 11:48:09.256 4105 INFO heat.engine.scheduler [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Task create from ResourceGroup "swarm_masters" Stack "swarm-cluster_test-nftqb3b77za6" [89003fb7-a094-4ed8-b744-00121e8cd577] timed out
2017-01-10 11:48:09.286 4102 INFO heat.engine.service [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Starting cancel of updating stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws
2017-01-10 11:48:09.336 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Stack CREATE cancelled
2017-01-10 11:48:09.383 4105 INFO heat.engine.stack [req-d44de78e-2835-43ad-8669-bfeaa96ccb20 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6): Timed out
2017-01-10 11:48:09.405 4102 INFO heat.engine.stack [req-d0e8b866-3e6e-44f0-a9f3-b2be5c40a981 641b42e296d544fdbf9fe7e8bfc65c57 282838c02c784f7ab8c89dd37ccfa87c - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 INFO heat.engine.resource [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] CREATE: TemplateResource "0" [eeebab48-bfe8-449c-a39f-341d99e88074] Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws" [10655a90-845e-4b7b-8fce-2f7e7a97ae15]
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource Traceback (most recent call last):
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     yield
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 855, in _do_action
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     step = next(subtask)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 806, in action_handler_task
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     done = check(handler_data)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 355, in check_create_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     return self._check_status_complete(self.CREATE)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 405, in _check_status_complete
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource     action=action)
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource ResourceFailure: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.407 4108 ERROR heat.engine.resource
2017-01-10 11:48:10.479 4108 INFO heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws): Resource CREATE failed: resources[0]: Stack CREATE cancelled
2017-01-10 11:48:10.488 4108 WARNING heat.engine.stack [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Failed to set state of stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws with traversal ID 21567a10-300d-4119-b1a1-f140792a6b0c, to CREATE_FAILED
2017-01-10 11:48:10.510 4108 INFO heat.engine.check_resource [req-a965d213-cadb-48bf-b836-5e657084f073 - - - - -] Re-trigger resource: (68, True)
2017-01-10 12:17:48.436 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:48.457 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_CloudWatch_Alarm.yaml
2017-01-10 12:17:53.246 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:17:53.256 4101 INFO heat.common.urlfetch [req-3c03ad0e-eb38-4376-977f-65b6bc364617 102efd10ea9342eebd00ae5a606bcf0c 9f9590d0c9eb47aba9f282b523efbb7f - - -] Fetching data from file:///etc/heat/templates/AWS_RDS_DBInstance.yaml
2017-01-10 12:29:31.935 4085 INFO heat.engine.resources.openstack.heat.wait_condition [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074] Timed out (0 of 1 received)
2017-01-10 12:29:31.936 4085 INFO heat.engine.resource [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] CREATE: HeatWaitCondition "master_wait_condition" Stack "swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52" [eeebab48-bfe8-449c-a39f-341d99e88074]
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource Traceback (most recent call last):
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 753, in _action_recorder
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 855, in _do_action
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 353, in wrapper
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     step = next(subtask)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 806, in action_handler_task
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     done = check(handler_data)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 130, in check_create_complete
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     return self._wait(*data)
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/wait_condition.py", line 108, in _wait
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource     raise exc
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource WaitConditionTimeout: 0 of 1 received
2017-01-10 12:29:31.936 4085 ERROR heat.engine.resource
2017-01-10 12:29:31.999 4085 INFO heat.engine.stack [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Stack CREATE FAILED (swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52): Resource CREATE failed: WaitConditionTimeout: resources.master_wait_condition: 0 of 1 received
2017-01-10 12:29:32.014 4085 WARNING heat.engine.stack [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Failed to set state of stack swarm-cluster_test-nftqb3b77za6-swarm_masters-ukvpffafljws-0-v2vaqnbd3h52 with traversal ID e5ef9bd4-8eab-4bec-bd42-d58b1a66828f, to CREATE_FAILED
2017-01-10 12:29:32.039 4085 INFO heat.engine.check_resource [req-8ccd805a-f0ff-4080-8585-d2cd65929ca0 - - - - -] Re-trigger resource: (71, True)

I am unable to find out what’s wrong.

Thanks in advance.

Any help will be appreciated.