Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Controller Node out of sync, can't add it back to environment

Hi Everyone, I am pretty new to the world of openstack and find myself with a bit of a problem. I noticed that one of our nodes was not connecting at the right speed so decided to try to just power it off and back on and see what happened. Once I did this, I am told this resulted in out of sync databases. Unable to fix the problem I dropped the controller node from the environment in fuel and tried to add it back but keep getting this error:

2016-09-13 19:31:45 ERR  (/Stage[main]/Galera/Exec[wait-for-synced-state]/returns) change from notrun to 0
failed: /usr/bin/mysql -uwsrep_sst -phTUuaexv -Nbe "show status like 'wsrep_local_state_comment'" | /bin/grep -q Synced && sleep 10 returned 1 instead of one of [0]
 2016-09-13 19:31:45    ERR  /usr/bin/puppet:4:in `<main>'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'  
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
 2016-09-13 19:31:45    ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:142:in `sync'
2016-09-13 19:31:45 ERR  /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'

Would anyone have any suggestions as to what I should try next?