Tripleo overcloud deployment fails during AllNodesDeploySteps resource creation
I've been trying to install the tripleo/newton environment in my home lab for the past week and my overcloud deployment keeps failing during creation of AllNodesDeploySteps resource step. I am looking for help trying to debug the root cause of this failure;
[stack@instack ~]$ openstack stack resource list overcloud | egrep -v COMPLETE +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+ | AllNodesDeploySteps | 44f774c8-b8b0-409f-ba84-1848675dbff8 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-01-29T17:08:00Z | | AllNodesExtraConfig | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | OS::TripleO::AllNodesExtraConfig | CREATE_FAILED | 2017-01-29T17:08:00Z | +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+
Below is relevant CLI command ouput that I've used to diagnose the failure. I've highlighted in bold what I believe to be the fatal error:
Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::vip_hosts for overcloud-novacompute-0.localdomain at /var/lib/heat-config/heat-config-puppet/82d68fda-3dbf-4294-8acc-bd19e014f52f.pp:51:1 on node overcloud-novacompute-0.localdomain
I can't seem to find any details on the function class :tripleo::vip_hosts and I am not sure what the next step should be to further debug this condition. I've resolved all errors as reported by node introspection. If somebody can confirm my diagnosis is correct and possible help with resolution I would be greatly appreciative. I've spent many many hours trying to get this to work!!
[stack@instack ~]$ heat stack-list --show-nested -f "status=FAILED" WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+ | id | stack_name | stack_status | creation_time | updated _time | parent | +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+ | 7995d855-c51b-471d-a585-7376b933a38b | overcloud | CREATE_FAILED | 2017-01-29T17:08:00Z | None | None | | 44f774c8-b8b0-409f-ba84-1848675dbff8 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | CREATE_FAILED | 2017-01-29T17:15:16Z | None | 7995d855-c51b-471d-a585-7376b933a38b | | 092d4164-1b35-43db-93eb-a6dbeb3cd501 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ControllerArtifactsDeploy-7lhxkq4hganb | CREATE_FAILED | 2017-01-29T17:15:27Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | daed916f-abf4-4c41-8693-eef9a03977b5 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk | CREATE_FAILED | 2017-01-29T17:15:57Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | 3f665e23-d6c1-4e33-8055-745a768e427e | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ComputeDeployment_Step1-xdklemxlx5xm | CREATE_FAILED | 2017-01-29T17:15:59Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 7995d855-c51b-471d-a585-7376b933a38b | | 9ca87aa2-b9e9-4016-a19d-c0ffeed4ca28 | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-ComputeSwapDeployment-m7h35oallirs | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | | da2ae5f6-cdfa-4e1a-b361-66398b1df2ab | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-CephStorageSwapDeployment-xaw4uhmpb2a3 | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | | 280aa462-f9fb-4fbf-a571-5d50779ba5ed | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-ControllerSwapDeployment-mbjbqbwo7b2w | CREATE_FAILED | 2017-01-29T17:16:13Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+
[stack@instack ~]$ heat resource-list --nested-depth 5 overcloud | grep FAILED WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead | AllNodesDeploySteps | 44f774c8-b8b0-409f-ba84-1848675dbff8 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-01-29T17:08:00Z | overcloud | | AllNodesExtraConfig | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | OS::TripleO::AllNodesExtraConfig | CREATE_FAILED | 2017-01-29T17:08:00Z | overcloud | | CephStorageDeployment_Step1 | daed916f-abf4-4c41-8693-eef9a03977b5 | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-01-29T17:15:18Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | ComputeDeployment_Step1 | 3f665e23-d6c1-4e33-8055-745a768e427e | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-01-29T17:15:19Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | ControllerArtifactsDeploy | 092d4164-1b35-43db-93eb-a6dbeb3cd501 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2017-01-29T17:15:19Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | 0 | 394b6851-f49e-488e-ae79-c1b45ba993b7 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-01-29T17:15:57Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk | | 0 | 34ef1259-f622-41af-82b8-bc530d8fb632 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-01-29T17:15:59Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ComputeDeployment_Step1-xdklemxlx5xm | | CephStorageSwapDeployment | da2ae5f6-cdfa-4e1a-b361-66398b1df2ab | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | | ComputeSwapDeployment | 9ca87aa2-b9e9-4016-a19d-c0ffeed4ca28 | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | | ControllerSwapDeployment | 280aa462-f9fb-4fbf-a571-5d50779ba5ed | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy |
[stack@instack ~]$ heat resource-show daed916f-abf4-4c41-8693-eef9a03977b5 0 WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "deploy_stdout": null, | | | "deploy_stderr": null, | | | "deploy_status_code": null | | | } | | creation_time | 2017-01-29T17:15:57Z | | description | | | links | http://192.0.2.1:8004/v1/b1082e04516c4956a607da1ab2dc77ec/stacks/overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk/daed916f-abf4-4c41-8693-eef9a03977b5/resources/0 (http://192.0.2.1:8004/v1/b1082e04516c...) (self) | | | http://192.0.2.1:8004/v1/b1082e04516c4956a607da1ab2dc77ec/stacks/overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk/daed916f-abf4-4c41-8693-eef9a03977b5 (http://192.0.2.1:8004/v1 ...