Ask Your Question
0

Savanna with Nova-network with Grizlly

asked 2013-08-06 10:36:02 -0500

sarita18narwal gravatar image

I had deployed the hadoop cluster using Savanna API. After launching the cluster it remains in waiting state and after some time goes into Eror State while the nodes remains in active state.

Suppose i had 3 node(1 master and 2 slave) in cluster . When I launched the cluster using respective cluster template,the scenario is like this.

Cluster

Name State Instance Count Sample Error 3

And in Instances

Name IP State 1-master 10.0.0.X ACTIVE 2-slave 10.0.0.X ACTIVE 3-slave 10.0.0.X ACTIVE

Log Description: WARNING savanna.service.instances [-] Can't start cluster 'tstcluster' (reason: Unauthorized (HTTP 401))

ERROR root [-] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 38, in create_cluster\n _await_instances(cluster)\n', ' File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 206, in _await_instances\n if not _check_if_up(instance):\n', ' File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 215, in _check_if_up\n server = instance.nova_info\n', ' File "/usr/local/lib/python2.7/dist-packages/savanna/db/models.py", line 226, in nova_info\n return nova.client().servers.get(self.instance_id)\n', ' File "/usr/local/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 350, in get\n return self._get("/servers/%s" % base.getid(server), "server")\n', ' File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 140, in _get\n _resp, body = self.api.client.get(url)\n', ' File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 230, in get\n return self._cs_request(url, \'GET\', *kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 227, in _cs_request\n raise e\n', 'Unauthorized: Unauthorized (HTTP 401)\n'] Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait readers.get(fileno, noop).cb(fileno) File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main result = function(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna/context.py", line 127, in wrapper func(args, *kwargs) File "/usr/local/lib/python2.7/dist-packages/savanna/service/api.py", line 111, in _provision_cluster i.create_cluster(cluster) File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 51, in create_cluster _rollback_cluster_creation(cluster, ex) File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 274, in _rollback_cluster_creation _shutdown_instances(cluster, True) File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 303, in _shutdown_instances _shutdown_instance(instance) File "/usr/local/lib/python2.7/dist-packages/savanna/service/instances.py", line 309, in _shutdown_instance nova.client().servers.delete(instance.instance_id) File "/usr/local/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 630, in delete self._delete("/servers/%s" % base.getid(server)) File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 154, in _delete _resp, _body = self.api.client.delete(url) File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py ... (more)

edit retag flag offensive close merge delete

29 answers

Sort by ยป oldest newest most voted
0

answered 2013-11-04 17:34:33 -0500

alazarev gravatar image

dikshith,

According to config you are using neutron (use_neutron=True) and doesn't use floating IPs (use_floating_ips=False). In this case host with Savanna need to have direct access to VMs via private network (usually it has not). You can always see IP address Savanna trying to use in cluster details (management IP column). Please check that you can ssh to VMs from the host with Savanna after VMs started up.

What version of savanna do you use? Master branch contains fix with netns proxy support ( https://review.openstack.org/#/c/52997/ ). Enabling it (use_namespaces=True) could help you with the issue.

edit flag offensive delete link more
0

answered 2013-11-04 13:54:21 -0500

I am facing the same problem . I am using savanna 0.3 and neutron network. The cluster is in waiting state for 24 hours after which the keystone token is invalid. Sometimes i can log in to the instances. But HDFS is not working in the system. Below is the hadoop report log hadoop dfsadmin -report report: FileSystem file:/// is not a distributed file system Usage: java DFSAdmin [-report]

Here is my savanna configuration file

host="localhost" port=8386

Address and credentials that will be used to check auth tokens

os_auth_host=10.2.1.3 os_auth_port=5000 os_admin_username=admin os_admin_password=*** os_admin_tenant_name=admin

use_floating_ips=False

use_neutron=True

debug=true

verbose=true

plugins=vanilla,hdp

[plugin:vanilla] plugin_class=savanna.plugins.vanilla.plugin:VanillaProvider

[plugin:hdp] plugin_class=savanna.plugins.hdp.ambariplugin:AmbariPlugin

[database] connection=sqlite:///savanna.sqlite

edit flag offensive delete link more
0

answered 2013-08-16 10:24:09 -0500

sarita18narwal gravatar image

Alexander Rubstsov ,

Thanks a lot. Thanks for listening me patiently and solving my problem.

My Cluster is now in ACTIVE state and i am unable to access my instance using web UI. :) :) :) :)

edit flag offensive delete link more
0

answered 2013-08-16 09:32:00 -0500

You should enable floating ip assignment in your OpenStack environment ("auto_assign_floating_ip=True" in the nova.conf) Please make sure that Savanna host can reach instances through floating ip.

edit flag offensive delete link more
0

answered 2013-08-16 08:39:11 -0500

sarita18narwal gravatar image

What to do if I have to use floating ip as managed ip .

when I have mention the option using_floating_ips=True,cluster goes into waiting state. So what to do for that to get rid out of it ?

edit flag offensive delete link more
0

answered 2013-08-16 06:39:36 -0500

Sarita,

You can access to instances using ssh-key that you specified when you create a cluster. Using Swift is optional. You can don't use it if you want.

edit flag offensive delete link more
0

answered 2013-08-16 06:26:57 -0500

sarita18narwal gravatar image

Alexander,

I did the changes in savanna.conf (use_floating_ips=False) and launch a new cluster named - Cluster-2.

Now my cluster is in ACTIVE State.

Thanks a lot for patience and helping me out . :) :) :) :) :)

Can u please help me about the access of the instances. What is the password of that image for username ubuntu i.e. http://savanna-files.mirantis.com/savanna-0.2-vanilla-1.1.2-ubuntu-12.10.qcow2 (http://savanna-files.mirantis.com/sav...)

and I am also unable to access the instance when i am using Web UI for MAP reduce and HDFS.

Is it necessary to use swift.?

edit flag offensive delete link more
0

answered 2013-08-16 06:25:52 -0500

sarita18narwal gravatar image

Thanks Alexander Rubtsov, that solved my question.

edit flag offensive delete link more
0

answered 2013-08-16 05:53:56 -0500

Sarita,

since you can get access from Savanna host to instances through internal IP - change in the savanna.conf: use_floating_ips=False

edit flag offensive delete link more
0

answered 2013-08-16 03:38:58 -0500

sarita18narwal gravatar image

The savanna log for cluster created using REST is from cluster-1.

Please ignore the previous log for "cluster-tt."

Sorry for pasting some irrelevant information in log file description.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-08-06 10:36:02 -0500

Seen: 258 times

Last updated: Nov 04 '13