can not start cluster

asked 2013-08-09 08:09:35 -0500

jfzhang1984
1 ●1 ●1 ●1

when I start a cluster, sometimes it can be turn to active state, but sometimes it throws an error as follows:

2013-08-09 10:35:53.936 23506 WARNING savanna.service.instances [-] Can't start cluster 'asd' (reason: [Errno 111] ECONNREFUSED) 2013-08-09 10:35:53.939 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs 2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs 2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs 2013-08-09 10:35:53.942 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/hubs/poll.py", line 99, in wait writers.get(fileno, noop).cb(fileno) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenthread.py", line 194, in main result = function(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/context.py", line 127, in wrapper func(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/service/api.py", line 115, in _provision_cluster plugin.configure_cluster(cluster) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 81, in configure_cluster self._push_configs_to_nodes(cluster) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 192, in _push_configs_to_nodes 'sudo chown -R $USER:$USER /etc/hadoop' File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 142, in execute_command return execute_command(self.ssh_connection(), cmd) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 133, in ssh_connection self._ssh = self.helper.ssh_connection() File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 96, in ssh_connection self.instance.node_group.cluster.private_key) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 31, in setup_ssh_connection ssh.connect(host, username=username, pkey=private_key) File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in connect retry_on_signal(lambda: sock.connect(addr)) File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/util.py", line 278, in retry_on_signal return function() File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in <lambda> retry_on_signal(lambda: sock.connect(addr)) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 194, in connect socket_checkerr(fd) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 46, in socket_checkerr raise socket.error(err, errno.errorcode[err]) error: [Errno 111] ECONNREFUSED Removing descriptor: 5

edit retag flag offensive close merge delete

add a comment

12 answers

Sort by » oldest newest most voted

answered 2014-02-07 05:21:32 -0500

alazarev
181 ●1 ●6

Hi Gowri,

Could you please provide savanna stacktrace on this error?

edit flag offensive delete link

add a comment

answered 2013-10-10 11:24:52 -0500

gzsgwsgwl
1 ●1 ●2 ●2

about the above problem, i add a detail log.

2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] Ciphers agreed: local=aes128-ctr, remote=aes128-ctr _log /opt/stack/savanna-venv/local/lib/python2.7/site-packag es/paramiko/transport.py:1406 2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.348 9109 DEBUG paramiko.transport [-] Switch to new keys ... _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Adding ssh-rsa host key for 192.168.64.61: 62220395651f2be9f614196bef3c3a08 _log /opt/stack/savanna-venv/local/l ib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Trying SSH key b06756077a580e0c84aa233e4e484ad8 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/p aramiko/transport.py:1406 2013-10-10 19:19:53.388 9109 DEBUG paramiko.transport [-] userauth is OK _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.476 9109 INFO paramiko.transport [-] Authentication (publickey) successful! 2013-10-10 19:19:53.479 9109 DEBUG paramiko.transport [-] [chan 1] Max packet in: 34816 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan nel.py:1115 2013-10-10 19:19:53.784 9109 DEBUG paramiko.transport [-] [chan 1] Max packet out: 32768 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/cha nnel.py:1115 2013-10-10 19:19:53.784 9109 INFO paramiko.transport [-] Secsh channel 1 opened. 2013-10-10 19:19:53.788 9109 DEBUG paramiko.transport [-] [chan 1] Sesch channel 1 request ok _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan nel.py:1115 2013-10-10 19:19:53.792 9109 INFO paramiko.transport.sftp [-] [chan 1] Opened sftp connection (server version 3) 2013-10-10 19:19:53.793 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/parami ko/sftp.py:132 2013-10-10 19:19:53.794 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-pac kages/paramiko/sftp.py:132 2013-10-10 19:19:53.795 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13 2 2013-10-10 19:19:53.797 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.p y:132 2013-10-10 19:19:53.798 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/par amiko/sftp.py:132 2013-10-10 19:19:53.799 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13 2 ... (more)

edit flag offensive delete link

add a comment

answered 2013-10-11 01:50:50 -0500

gzsgwsgwl
1 ●1 ●2 ●2

for this question, there is also situation that every thing will be normal when i start in debug mode in the course of unormal ssh connection .

edit flag offensive delete link

add a comment

answered 2013-08-30 10:20:22 -0500

ruhe
1

You don't need to add this argument. virtual environment prevents conflicts with existing python packages installed on your system.

you also can install savanna on a in your system path using the following command: $ pip install savanna

edit flag offensive delete link

add a comment

answered 2014-02-07 05:02:46 -0500

gouribudal
1

Hi gws, I am having a similar situation, I am using Mirantis Fuel 4.0, I used image[centos-6-64-hdp-vanilla.qcow2 ] from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html] (http://savanna.readthedocs.org/en/lat...) . The inastances are up and I can SSH and ping them but the cluster turns into erroe state .

Savanna logs have something like ' failed execution chown $USER:$USER /etc/hadoop ,RuntimeError : node newcluster_vanilla_worker-001 has error status '.

Do you have any idea about this? I am stuck with this for quite some time :(.Since I am using Fuel installer, it is supposed to take care of all configuration after ti launch cluster using a registered image.Any help is appreciated.

Thanks.

edit flag offensive delete link

add a comment

answered 2013-08-12 09:29:34 -0500

arubtsov
1

Hello jianfeng2013,

Did you install Savanna in a separate python virtual environment? In order to do that run: "cd ~; virtualenv savanna-venv" This will install new virtual environment into savanna-venv directory in your home directory.

Then you can install Savanna inside this virtual environment: "savanna-venv/bin/pip install savanna" (for more information please visit http://savanna.readthedocs.org/en/latest/userdoc/installation.guide.html (http://savanna.readthedocs.org/en/lat...) )

After that try to create cluster again

edit flag offensive delete link

add a comment

answered 2013-08-30 03:10:53 -0500

jfzhang1984
1 ●1 ●1 ●1

I want to known, Does savanna must be installed in the virtual enviroment?

When I create the virtal enviroment, should i assign the argument --no-site-packages ?

edit flag offensive delete link

add a comment

answered 2013-10-10 10:59:06 -0500

gzsgwsgwl
1 ●1 ●2 ●2

i encounter the same problem,sometimes start cluster successfully displaying active,sometimes display "File "/opt/stack/savanna/local/lib/python2.7/site-packages/eventlet/greenio.py", line 46, in socket_checkerr raise socket.error(err, errno.errorcode[err]) error: [Errno 111] ECONNREFUSED"

can someone know whether this is a bug?

edit flag offensive delete link

add a comment

answered 2014-02-07 06:04:22 -0500

gouribudal
1

Hi Andrew Lazarev,

1.The trace is as follows:

2014-02-05 14:15:24.518 19770 WARNING savanna.service.instances [-] Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status) 2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Error 2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster 'ostf-test-savanna-cluster' creation rollback (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status) 2014-02-05 14:15:24.618 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-05 14:15:24.913 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-05 14:15:25.214 19770 ERROR savanna.context [-] Thread 'cluster-creating-3d244250-d9cb-4a32-bbc5-91159416b960' fails with exception: 'node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status' 2014-02-05 14:15:25.214 19770 TRACE savanna.context Traceback (most recent call last): 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper 2014-02-05 14:15:25.214 19770 TRACE savanna.context func(args, *kwargs) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 153, in _provision_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context i.create_cluster(cluster) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 75, in create_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context _rollback_cluster_creation(cluster, ex) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/openstack/common/excutils.py", line 70, in __exit__ 2014-02-05 14:15:25.214 19770 TRACE savanna.context six.reraise(self.type_, self.value, self.tb) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 51, in create_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context _await_active(cluster, instances) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 342, in _await_active 2014-02-05 14:15:25.214 19770 TRACE savanna.context if _check_if_active(instance): 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 355, in _check_if_active 2014-02-05 14:15:25.214 19770 TRACE savanna.context raise RuntimeError("node %s has error status" % server.name) 2014-02-05 14:15:25.214 19770 TRACE savanna.context RuntimeError: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status 2014-02-05 14:15:25.214 19770 TRACE savanna.context 2014-02-05 14:15:38.165 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Deleting 2014-02-05 14:18:17.791 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.30.0.3 2014-02-07 04:26:21.673 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-07 04:26:22.106 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Validating 2014-02-07 04:26:22.215 ... (more)

edit flag offensive delete link

add a comment

answered 2014-02-07 21:09:02 -0500

alazarev
181 ●1 ●6

"Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status)" This means that instance didn't appear in 'active' state. You should check openstack logs on the reason.
"'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"" I see you are using vanilla plugin while both images you listed are for HDP plugin. Please use only images supported by plugin you use.

edit flag offensive delete link

add a comment

can not start cluster

12 answers

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

can not start cluster edit

12 answers

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

can not start cluster