Sahara:Can't login to nodes

asked 2014-12-23 03:12:51 -0500

BrightLiao gravatar image

I can successfully create a node template, cluster template, and cluster in Sahara through horizon. However when I attempt to launch the cluster, it is "waiting" all the time.

1) Sahara lives on the controller node

2) We use Neutron to support the network of clusters:

use_neutron=True

use_namespaces=True

3) We don't have enough floating_ips for each instance, so Sahara can't use floating_ips to access instances:

use_floating_ip=Flase

The logs state it can't access the instances:

2014-12-23 16:51:23.028 483 DEBUG sahara.utils.ssh_remote [-] [test-master-new-001] Executing "ls .ssh/authorized_keys" _log_command /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:407

2014-12-23 16:51:23.043 483 DEBUG sahara.utils.ssh_remote [-] Returning neutron info: {'host': u'10.0.2.2', 'token': u'311bdf137f83e54fae0a76459e7868ce', 'network': u'50ec6fbd-dea2-4401-a375-c166a383e0f4', 'tenant': u'admin', 'uri': u'http://202.38.95.145:9696'} get_neutron_info /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:306

2014-12-23 16:51:23.212 483 DEBUG sahara.utils.ssh_remote [-] [test-slave-new-001] Executing "ls .ssh/authorized_keys" _log_command /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:407

2014-12-23 16:51:23.232 483 DEBUG sahara.utils.ssh_remote [-] Returning neutron info: {'host': u'10.0.2.5', 'token': u'311bdf137f83e54fae0a76459e7868ce', 'network': u'50ec6fbd-dea2-4401-a375-c166a383e0f4', 'tenant': u'admin', 'uri': u'http://202.38.95.145:9696'} get_neutron_info /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:306

2014-12-23 16:51:23.548 483 DEBUG sahara.utils.ssh_remote [-] [test-master-new-001] _execute_command took 0.5 seconds to complete _log_command /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:407

2014-12-23 16:51:23.549 483 DEBUG sahara.service.engine [-] Can't login to node test-master-new-001 (10.0.2.2), reason ProxyCommandFailure: "ProxyCommand (ip netns exec qrouter-4bf3a028-2073-4ea0-8d71-6d02fbebfa50 nc 10.0.2.2 22)" returned non-zero exit status: Broken pipe _wait_until_accessible /root/sahara-venv/local/lib/python2.7/site-packages/sahara/service/engine.py:95

2014-12-23 16:51:23.739 483 DEBUG sahara.utils.ssh_remote [-] [test-slave-new-001] _execute_command took 0.5 seconds to complete _log_command /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:407

2014-12-23 16:51:23.740 483 DEBUG sahara.service.engine [-] Can't login to node test-slave-new-001 (10.0.2.5), reason ProxyCommandFailure: "ProxyCommand (ip netns exec qrouter-4bf3a028-2073-4ea0-8d71-6d02fbebfa50 nc 10.0.2.5 22)" returned non-zero exit status: Broken pipe _wait_until_accessible /root/sahara-venv/local/lib/python2.7/site-packages/sahara/service/engine.py:95

2014-12-23 16:51:24.139 483 DEBUG sahara.utils.ssh_remote [-] [test-slave-new-002] Executing "ls .ssh/authorized_keys" _log_command /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:407

2014-12-23 16:51:24.158 483 DEBUG sahara.utils.ssh_remote [-] Returning neutron info: {'host': u'10.0.2.6', 'token': u'311bdf137f83e54fae0a76459e7868ce', 'network': u'50ec6fbd-dea2-4401-a375-c166a383e0f4', 'tenant': u'admin', 'uri': u'http://202.38.95.145:9696'} get_neutron_info /root/sahara-venv/local/lib/python2.7/site-packages/sahara/utils/ssh_remote.py:306

2014-12-23 16:51:24.660 483 DEBUG sahara.utils.ssh_remote [-] [test-slave-new-002] _execute_command took 0.5 seconds to complete _log_command /root/sahara-venv/local/lib ...

(more)
edit retag flag offensive close merge delete

Comments

have checked the security group is opening for port 22 or not?

9lives gravatar image9lives ( 2014-12-23 03:48:30 -0500 )edit

Yes, port 22 of intances is accessable. We can ssh to intance from other hosts.

BrightLiao gravatar imageBrightLiao ( 2014-12-24 03:06:38 -0500 )edit

can you execute the above command manually and see what happened?

ip netns exec qrouter-4bf3a028-2073-4ea0-8d71-6d02fbebfa50 nc 10.0.2.6 22

besides, what did your ovs-vsctl show say?

9lives gravatar image9lives ( 2014-12-24 03:19:25 -0500 )edit

I executed the command "ip netns ..." , it shows : "Cannot open network namespace: No such file or directory."

and ovs-vsctl show : 18829f8c-b4c7-4e3f-844b-cb06f41e5a51 ovs_version: "1.10.2"

BrightLiao gravatar imageBrightLiao ( 2014-12-25 03:12:37 -0500 )edit

Namespace of OpenStack is in network node(Neutron) while Sahara runs on controller node. Is it the reason of "Cannot open network namespace: No such file or directory" ?

BrightLiao gravatar imageBrightLiao ( 2014-12-25 03:35:57 -0500 )edit