Ask Your Question
0

can not start cluster

asked 2013-08-09 08:09:35 -0500

jfzhang1984 gravatar image

when I start a cluster, sometimes it can be turn to active state, but sometimes it throws an error as follows:

2013-08-09 10:35:53.936 23506 WARNING savanna.service.instances [-] Can't start cluster 'asd' (reason: [Errno 111] ECONNREFUSED) 2013-08-09 10:35:53.939 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs 2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs 2013-08-09 10:35:53.941 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' for xml configs 2013-08-09 10:35:53.942 23506 WARNING savanna.plugins.vanilla.config_helper [-] Plugin recieved wrong applicable target 'general' in environmental configs Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/hubs/poll.py", line 99, in wait writers.get(fileno, noop).cb(fileno) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenthread.py", line 194, in main result = function(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/context.py", line 127, in wrapper func(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/service/api.py", line 115, in _provision_cluster plugin.configure_cluster(cluster) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 81, in configure_cluster self._push_configs_to_nodes(cluster) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/plugins/vanilla/plugin.py", line 192, in _push_configs_to_nodes 'sudo chown -R $USER:$USER /etc/hadoop' File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 142, in execute_command return execute_command(self.ssh_connection(), cmd) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 133, in ssh_connection self._ssh = self.helper.ssh_connection() File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 96, in ssh_connection self.instance.node_group.cluster.private_key) File "/usr/local/lib/python2.7/dist-packages/savanna-0.2.1-py2.7.egg/savanna/utils/remote.py", line 31, in setup_ssh_connection ssh.connect(host, username=username, pkey=private_key) File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in connect retry_on_signal(lambda: sock.connect(addr)) File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/util.py", line 278, in retry_on_signal return function() File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/client.py", line 305, in <lambda> retry_on_signal(lambda: sock.connect(addr)) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 194, in connect socket_checkerr(fd) File "/usr/local/lib/python2.7/dist-packages/eventlet-0.13.0-py2.7.egg/eventlet/greenio.py", line 46, in socket_checkerr raise socket.error(err, errno.errorcode[err]) error: [Errno 111] ECONNREFUSED Removing descriptor: 5

edit retag flag offensive close merge delete
add a comment

12 answers

Sort by ยป oldest newest most voted
12next »
0

answered 2014-02-07 05:21:32 -0500

alazarev gravatar image

Hi Gowri,

Could you please provide savanna stacktrace on this error?

edit flag offensive delete link more
add a comment
0

answered 2013-10-10 11:24:52 -0500

gzsgwsgwl gravatar image

about the above problem, i add a detail log.

2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] Ciphers agreed: local=aes128-ctr, remote=aes128-ctr _log /opt/stack/savanna-venv/local/lib/python2.7/site-packag es/paramiko/transport.py:1406 2013-10-10 19:19:53.320 9109 DEBUG paramiko.transport [-] using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.348 9109 DEBUG paramiko.transport [-] Switch to new keys ... _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Adding ssh-rsa host key for 192.168.64.61: 62220395651f2be9f614196bef3c3a08 _log /opt/stack/savanna-venv/local/l ib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.349 9109 DEBUG paramiko.transport [-] Trying SSH key b06756077a580e0c84aa233e4e484ad8 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/p aramiko/transport.py:1406 2013-10-10 19:19:53.388 9109 DEBUG paramiko.transport [-] userauth is OK _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/transport.py:1406 2013-10-10 19:19:53.476 9109 INFO paramiko.transport [-] Authentication (publickey) successful! 2013-10-10 19:19:53.479 9109 DEBUG paramiko.transport [-] [chan 1] Max packet in: 34816 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan nel.py:1115 2013-10-10 19:19:53.784 9109 DEBUG paramiko.transport [-] [chan 1] Max packet out: 32768 bytes _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/cha nnel.py:1115 2013-10-10 19:19:53.784 9109 INFO paramiko.transport [-] Secsh channel 1 opened. 2013-10-10 19:19:53.788 9109 DEBUG paramiko.transport [-] [chan 1] Sesch channel 1 request ok _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/chan nel.py:1115 2013-10-10 19:19:53.792 9109 INFO paramiko.transport.sftp [-] [chan 1] Opened sftp connection (server version 3) 2013-10-10 19:19:53.793 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/parami ko/sftp.py:132 2013-10-10 19:19:53.794 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('authorized_keys', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-pac kages/paramiko/sftp.py:132 2013-10-10 19:19:53.795 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13 2 2013-10-10 19:19:53.797 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.p y:132 2013-10-10 19:19:53.798 9109 DEBUG paramiko.transport.sftp [-] [chan 1] open('id_rsa', 'w') -> 00000000 _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/par amiko/sftp.py:132 2013-10-10 19:19:53.799 9109 DEBUG paramiko.transport.sftp [-] [chan 1] close(00000000) _log /opt/stack/savanna-venv/local/lib/python2.7/site-packages/paramiko/sftp.py:13 2 ... (more)

edit flag offensive delete link more
add a comment
0

answered 2013-10-11 01:50:50 -0500

gzsgwsgwl gravatar image

for this question, there is also situation that every thing will be normal when i start in debug mode in the course of unormal ssh connection .

edit flag offensive delete link more
add a comment
0

answered 2013-08-30 10:20:22 -0500

ruhe gravatar image

You don't need to add this argument. virtual environment prevents conflicts with existing python packages installed on your system.

you also can install savanna on a in your system path using the following command: $ pip install savanna

edit flag offensive delete link more
add a comment
0

answered 2014-02-07 05:02:46 -0500

gouribudal gravatar image

Hi gws, I am having a similar situation, I am using Mirantis Fuel 4.0, I used image[centos-6-64-hdp-vanilla.qcow2 ] from [link : http://savanna.readthedocs.org/en/latest/userdoc/hdp_plugin.html] (http://savanna.readthedocs.org/en/lat...) . The inastances are up and I can SSH and ping them but the cluster turns into erroe state .

Savanna logs have something like ' failed execution chown $USER:$USER /etc/hadoop ,RuntimeError : node newcluster_vanilla_worker-001 has error status '.

Do you have any idea about this? I am stuck with this for quite some time :(.Since I am using Fuel installer, it is supposed to take care of all configuration after ti launch cluster using a registered image.Any help is appreciated.

Thanks.

edit flag offensive delete link more
add a comment
0

answered 2013-08-12 09:29:34 -0500

arubtsov gravatar image

Hello jianfeng2013,

Did you install Savanna in a separate python virtual environment? In order to do that run: "cd ~; virtualenv savanna-venv" This will install new virtual environment into savanna-venv directory in your home directory.

Then you can install Savanna inside this virtual environment: "savanna-venv/bin/pip install savanna" (for more information please visit http://savanna.readthedocs.org/en/latest/userdoc/installation.guide.html (http://savanna.readthedocs.org/en/lat...) )

After that try to create cluster again

edit flag offensive delete link more
add a comment
0

answered 2013-08-30 03:10:53 -0500

jfzhang1984 gravatar image

I want to known, Does savanna must be installed in the virtual enviroment?

When I create the virtal enviroment, should i assign the argument --no-site-packages ?

edit flag offensive delete link more
add a comment
0

answered 2013-10-10 10:59:06 -0500

gzsgwsgwl gravatar image

i encounter the same problem,sometimes start cluster successfully displaying active,sometimes display "File "/opt/stack/savanna/local/lib/python2.7/site-packages/eventlet/greenio.py", line 46, in socket_checkerr raise socket.error(err, errno.errorcode[err]) error: [Errno 111] ECONNREFUSED"

can someone know whether this is a bug?

edit flag offensive delete link more
add a comment
0

answered 2014-02-07 06:04:22 -0500

gouribudal gravatar image

Hi Andrew Lazarev,

1.The trace is as follows:

2014-02-05 14:15:24.518 19770 WARNING savanna.service.instances [-] Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status) 2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Error 2014-02-05 14:15:24.586 19770 INFO savanna.service.instances [-] Cluster 'ostf-test-savanna-cluster' creation rollback (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status) 2014-02-05 14:15:24.618 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-05 14:15:24.913 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-05 14:15:25.214 19770 ERROR savanna.context [-] Thread 'cluster-creating-3d244250-d9cb-4a32-bbc5-91159416b960' fails with exception: 'node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status' 2014-02-05 14:15:25.214 19770 TRACE savanna.context Traceback (most recent call last): 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/context.py", line 120, in _wrapper 2014-02-05 14:15:25.214 19770 TRACE savanna.context func(args, *kwargs) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/api.py", line 153, in _provision_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context i.create_cluster(cluster) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 75, in create_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context _rollback_cluster_creation(cluster, ex) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/openstack/common/excutils.py", line 70, in __exit__ 2014-02-05 14:15:25.214 19770 TRACE savanna.context six.reraise(self.type_, self.value, self.tb) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 51, in create_cluster 2014-02-05 14:15:25.214 19770 TRACE savanna.context _await_active(cluster, instances) 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 342, in _await_active 2014-02-05 14:15:25.214 19770 TRACE savanna.context if _check_if_active(instance): 2014-02-05 14:15:25.214 19770 TRACE savanna.context File "/usr/lib/python2.6/site-packages/savanna/service/instances.py", line 355, in _check_if_active 2014-02-05 14:15:25.214 19770 TRACE savanna.context raise RuntimeError("node %s has error status" % server.name) 2014-02-05 14:15:25.214 19770 TRACE savanna.context RuntimeError: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status 2014-02-05 14:15:25.214 19770 TRACE savanna.context 2014-02-05 14:15:38.165 19770 INFO savanna.service.api [-] Cluster status has been changed: id=3d244250-d9cb-4a32-bbc5-91159416b960, New status=Deleting 2014-02-05 14:18:17.791 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.30.0.3 2014-02-07 04:26:21.673 19770 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.16.8.13 2014-02-07 04:26:22.106 19770 INFO savanna.service.api [-] Cluster status has been changed: id=2da3e9b5-b19c-48f1-820f-472350a897e9, New status=Validating 2014-02-07 04:26:22.215 ... (more)

edit flag offensive delete link more
add a comment
0

answered 2014-02-07 21:09:02 -0500

alazarev gravatar image

  1. "Can't start cluster 'ostf-test-savanna-cluster' (reason: node ostf-test-savanna-cluster-ostf-test-worker-node-tt-dn-001 has error status)" This means that instance didn't appear in 'active' state. You should check openstack logs on the reason.

  2. "'RemoteCommandException: Error during command execution: "sudo chown -R $USER:$USER /etc/hadoop"" I see you are using vanilla plugin while both images you listed are for HDP plugin. Please use only images supported by plugin you use.

edit flag offensive delete link more
add a comment
12next »

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2013-08-09 08:09:35 -0500

Seen: 344 times

Last updated: Feb 19 '14

Related questions

Looking for job banaries examples link

Savanna with Nova-network with Grizlly

Java library to find if 2 CIDR's overlap

ETag of manifest file

Sahara: Master-instance: Service hadoop-* not running

Verify Keystone with Curl

Not able to create LBaaS pool

Add a new store

Problems scheduling across zones

Object upload timeout