Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Tripleo overcloud deployment fails during AllNodesDeploySteps resource creation

I've been trying to install the tripleo/newton environment in my home lab for the past week and my overcloud deployment keeps failing during creation of AllNodesDeploySteps resource step. I am looking for help trying to debug the root cause of this failure;

[stack@instack ~]$ openstack stack resource list overcloud | egrep -v COMPLETE +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+ | AllNodesDeploySteps | 44f774c8-b8b0-409f-ba84-1848675dbff8 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-01-29T17:08:00Z | | AllNodesExtraConfig | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | OS::TripleO::AllNodesExtraConfig | CREATE_FAILED | 2017-01-29T17:08:00Z | +-------------------------------------------+----------------------------------------------+-------------------------------------------------+-----------------+----------------------+

Below is relevant CLI command ouput that I've used to diagnose the failure. I've highlighted in bold what I believe to be the fatal error:

Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::vip_hosts for overcloud-novacompute-0.localdomain at /var/lib/heat-config/heat-config-puppet/82d68fda-3dbf-4294-8acc-bd19e014f52f.pp:51:1 on node overcloud-novacompute-0.localdomain

I can't seem to find any details on the function class :tripleo::vip_hosts and I am not sure what the next step should be to further debug this condition. I've resolved all errors as reported by node introspection. If somebody can confirm my diagnosis is correct and possible help with resolution I would be greatly appreciative. I've spent many many hours trying to get this to work!!

[stack@instack ~]$ heat stack-list --show-nested -f "status=FAILED" WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+ | id | stack_name | stack_status | creation_time | updated _time | parent | +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+ | 7995d855-c51b-471d-a585-7376b933a38b | overcloud | CREATE_FAILED | 2017-01-29T17:08:00Z | None | None | | 44f774c8-b8b0-409f-ba84-1848675dbff8 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | CREATE_FAILED | 2017-01-29T17:15:16Z | None | 7995d855-c51b-471d-a585-7376b933a38b | | 092d4164-1b35-43db-93eb-a6dbeb3cd501 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ControllerArtifactsDeploy-7lhxkq4hganb | CREATE_FAILED | 2017-01-29T17:15:27Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | daed916f-abf4-4c41-8693-eef9a03977b5 | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk | CREATE_FAILED | 2017-01-29T17:15:57Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | 3f665e23-d6c1-4e33-8055-745a768e427e | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ComputeDeployment_Step1-xdklemxlx5xm | CREATE_FAILED | 2017-01-29T17:15:59Z | None | 44f774c8-b8b0-409f-ba84-1848675dbff8 | | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 7995d855-c51b-471d-a585-7376b933a38b | | 9ca87aa2-b9e9-4016-a19d-c0ffeed4ca28 | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-ComputeSwapDeployment-m7h35oallirs | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | | da2ae5f6-cdfa-4e1a-b361-66398b1df2ab | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-CephStorageSwapDeployment-xaw4uhmpb2a3 | CREATE_FAILED | 2017-01-29T17:16:12Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | | 280aa462-f9fb-4fbf-a571-5d50779ba5ed | overcloud-AllNodesExtraConfig-4v22i6wyzvpy-ControllerSwapDeployment-mbjbqbwo7b2w | CREATE_FAILED | 2017-01-29T17:16:13Z | None | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | +--------------------------------------+-------------------------------------------------------------------------------------+---------------+----------------------+-------- ------+--------------------------------------+

[stack@instack ~]$ heat resource-list --nested-depth 5 overcloud | grep FAILED WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead | AllNodesDeploySteps | 44f774c8-b8b0-409f-ba84-1848675dbff8 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-01-29T17:08:00Z | overcloud | | AllNodesExtraConfig | 39bfdaaf-d0d2-4faf-9a3c-d89c012a4c50 | OS::TripleO::AllNodesExtraConfig | CREATE_FAILED | 2017-01-29T17:08:00Z | overcloud | | CephStorageDeployment_Step1 | daed916f-abf4-4c41-8693-eef9a03977b5 | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-01-29T17:15:18Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | ComputeDeployment_Step1 | 3f665e23-d6c1-4e33-8055-745a768e427e | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-01-29T17:15:19Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | ControllerArtifactsDeploy | 092d4164-1b35-43db-93eb-a6dbeb3cd501 | OS::Heat::StructuredDeployments | CREATE_FAILED | 2017-01-29T17:15:19Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at | | 0 | 394b6851-f49e-488e-ae79-c1b45ba993b7 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-01-29T17:15:57Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk | | 0 | 34ef1259-f622-41af-82b8-bc530d8fb632 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-01-29T17:15:59Z | overcloud-AllNodesDeploySteps-qy6hnc2uw4at-ComputeDeployment_Step1-xdklemxlx5xm | | CephStorageSwapDeployment | da2ae5f6-cdfa-4e1a-b361-66398b1df2ab | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | | ComputeSwapDeployment | 9ca87aa2-b9e9-4016-a19d-c0ffeed4ca28 | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy | | ControllerSwapDeployment | 280aa462-f9fb-4fbf-a571-5d50779ba5ed | OS::Heat::SoftwareDeploymentGroup | CREATE_FAILED | 2017-01-29T17:16:12Z | overcloud-AllNodesExtraConfig-4v22i6wyzvpy |

[stack@instack ~]$ heat resource-show daed916f-abf4-4c41-8693-eef9a03977b5 0 WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attributes | { | | | "deploy_stdout": null, | | | "deploy_stderr": null, | | | "deploy_status_code": null | | | } | | creation_time | 2017-01-29T17:15:57Z | | description | | | links | http://192.0.2.1:8004/v1/b1082e04516c4956a607da1ab2dc77ec/stacks/overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk/daed916f-abf4-4c41-8693-eef9a03977b5/resources/0 (self) | | | http://192.0.2.1:8004/v1/b1082e04516c4956a607da1ab2dc77ec/stacks/overcloud-AllNodesDeploySteps-qy6hnc2uw4at-CephStorageDeployment_Step1-s2jk5ep2hhvk/daed916f-abf4-4c41-8693-eef9a03977b5 (stack) | | logical_resource_id | 0 | | parent_resource | CephStorageDeployment_Step1 | | physical_resource_id | 394b6851-f49e-488e-ae79-c1b45ba993b7 | | required_by | | | resource_name | 0 | | resource_status | CREATE_FAILED | | resource_status_reason | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 | | resource_type | OS::Heat::StructuredDeployment | | updated_time | 2017-01-29T17:15:57Z | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[stack@instack ~]$ heat deployment-show 394b6851-f49e-488e-ae79-c1b45ba993b7 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "9631ac28-1a0e-47fd-ba8b-e6d66492254b", "config_id": "daf6e1e7-eb50-47a0-b495-4c58f71b359c", "output_values": { "deploy_stdout": "\u001b[mNotice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;33mWarning: Creating sysctl via Puppet.newtype is deprecated and will be removed in a future release. Use Puppet::Type.newtype instead.\n (at /usr/share/ruby/vendor_ruby/puppet.rb:162:in newtype')\u001b[0m\n\u001b[1;33mWarning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\n (file & line not available)\u001b[0m\n\u001b[1;33mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\"/var/lib/heat-config/heat-config-puppet/daf6e1e7-eb50-47a0-b495-4c58f71b359c.pp\", 32]:\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:indeprecation')\u001b[0m\n\u001b[1;31mError: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::vip_hosts for overcloud-cephstorage-0.localdomain at /var/lib/heat-config/heat-config-puppet/daf6e1e7-eb50-47a0-b495-4c58f71b359c.pp:37:1 on node overcloud-cephstorage-0.localdomain\u001b[0m\n", "deploy_status_code": 1 }, "creation_time": "2017-01-29T17:15:58Z", "updated_time": "2017-01-29T17:16:33Z", "input_values": { "step": 1, "update_identifier": "1485709657" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "394b6851-f49e-488e-ae79-c1b45ba993b7" }

[stack@instack ~]$ heat deployment-show 34ef1259-f622-41af-82b8-bc530d8fb632 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "ff449670-3955-4674-9856-54323edde8c7", "config_id": "82d68fda-3dbf-4294-8acc-bd19e014f52f", "output_values": { "deploy_stdout": "\u001b[mNotice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;33mWarning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\n (file & line not available)\u001b[0m\n\u001b[1;33mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\"/var/lib/heat-config/heat-config-puppet/82d68fda-3dbf-4294-8acc-bd19e014f52f.pp\", 30]:\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in deprecation')\u001b[0m\n\u001b[1;33mWarning: Creating sysctl via Puppet.newtype is deprecated and will be removed in a future release. Use Puppet::Type.newtype instead.\n (at /usr/share/ruby/vendor_ruby/puppet.rb:162:innewtype')\u001b[0m\n\u001b[1;31mError: Evaluation Error: Error while evaluating a Function Call, Could not find class ::tripleo::vip_hosts for overcloud-novacompute-0.localdomain at /var/lib/heat-config/heat-config-puppet/82d68fda-3dbf-4294-8acc-bd19e014f52f.pp:51:1 on node overcloud-novacompute-0.localdomain\u001b[0m\n", "deploy_status_code": 1 }, "creation_time": "2017-01-29T17:15:59Z", "updated_time": "2017-01-29T17:16:40Z", "input_values": { "step": 1, "update_identifier": "1485709657" }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "34ef1259-f622-41af-82b8-bc530d8fb632" }