Ask Your Question
1

Sahara: Master-instance: Service hadoop-* not running

asked 2014-07-14 11:10:37 -0500

datts gravatar image

updated 2014-07-15 11:15:00 -0500

smaffulli gravatar image

I install openstack icehouse, then install sahara. I create cluster, it worked! But when i login master-instance, test. Run hadoop jar hadoop-examples-1.2.1.jar pi 10 100 ==> Message error: java.io.IOException: Cannot create input directory PiEstimator_TMP_3_141592654/in...

I check service hadoop-namenode, it's not working. Message: start-stop-daemon: unable to set gid to 201 (Operation not permitted). All service hadoop-* not working, too.

Go back, i check log, have warning. Summarize:

WARNING sahara.service.engine [-] Can't start cluster '_unknown_' (reason: 'NoneType' object has no attribute 'node_groups')
WARNING sahara.service.engine [-] Presumably the operation failed because the cluster wasdeleted by a user during the process.
ERROR root [-] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/home/bkcloud/sahara-venv/local/lib/python2.7/site-packages/sahara/service/direct_engine.py", line 65, in create_cluster\n    volumes.attach(cluster)\n', '  File "/home/bkcloud/sahara-venv/local/lib/python2.7/site-packages/sahara/service/volumes.py", line 31, in attach\n    for node_group in cluster.node_groups:\n', "AttributeError: 'NoneType' object has no attribute 'node_groups'\n"]
ERROR sahara.context [-] Thread 'cluster-creating-91420b4b-9e89-44ff-8ce4-6cd5740e7bdc' fails with exception: 'Cluster id 'None' not found!'

Default hadoop located in /usr/share/hadoop, so if i execute sudo su hadoop; cd ~ anh run job, it's will not working

Not a valid JAR: /home/hadoop/hadoop-examples-1.2.1.jar

I use:

Details steps i created cluster: https://docs.google.com/document/d/1n...

edit retag flag offensive close merge delete

Comments

The info in the question is not full, but let me take a wild guess: if you execute sudo su hadoop; cd ~

and run Hadoop job after that, does the job work?

If that does not help, let me ask several questions: 1. Which plugin do you use: Vanilla or HDP? 2. Which Hadoop version do you install? 3. Sahara uses prepared images. Which image did you downloaded and used?

dmitrymex gravatar imagedmitrymex ( 2014-07-14 16:34:32 -0500 )edit

Edit your question to add more details, don't use comments to paste larger blobs of text (makes the question more difficult to understand).

smaffulli gravatar imagesmaffulli ( 2014-07-15 11:16:06 -0500 )edit

OK! Thank you reminded :)

datts gravatar imagedatts ( 2014-07-15 11:51:12 -0500 )edit

3 answers

Sort by ยป oldest newest most voted
0

answered 2014-07-14 16:13:03 -0500

smaffulli gravatar image

updated 2014-07-15 11:17:42 -0500

I would focus my attention on the java.io.IOException, it seems the most explicit one at the moment. Make sure that the wherever the temp dir is being created, your user has the appropriate permissions. Also, the error 'not a valid jar' makes me think that you may have not downloaded the correct files. Check those out, edit your question to add more details as you keep debugging.

edit flag offensive delete link more

Comments

Thank you!

I follow the instructions on the home page (section create cluster, i use UI Sahara): http://docs.openstack.org/developer/sahara/devref/quickstart.html (http://docs.openstack.org/developer/s...)

So, i think: I failed step config sahara or create node template? Because service sahara not running, so hadoop not worked :(

datts gravatar imagedatts ( 2014-07-14 23:03:12 -0500 )edit
0

answered 2014-07-15 11:31:47 -0500

dmitrymex gravatar image

Ok, you are doing alright. You just need to specify full path to the hadoop-examples.jar file, like that:

hadoop jar /usr/share/hadoop/<...>/hadoop-examples-1.2.1.jar pi 10 100

Don't forget to replace '<...>' in command I've proposed with the actual path, so that /usr/share/hadoop/<...>/hadoop-examples-1.2.1.jar a full path for hadoop-examples.jar.

edit flag offensive delete link more

Comments

Yes. But with file template sahara-icehouse-vanilla-1.2.1-ubuntu-13.10.qcow2, path /usr/share/hadoop/hadoop-examples-1.2.1.jar is true.

This my error is service hadoop-* not running, so hadoop not worked :(

datts gravatar imagedatts ( 2014-07-15 11:55:18 -0500 )edit

Did you try all the commands together:

sudo su hadoop
cd ~
hadoop jar /usr/share/hadoop/hadoop-examples-1.2.1.jar pi 10 100

I expect it to work because I think your problem was that you were running 'hadoop' command in catalog where you had no write permission. Also I think it is important to run it under 'hadoop' user.

dmitrymex gravatar imagedmitrymex ( 2014-07-15 14:55:52 -0500 )edit

Yes dmitrymex! Seems it worked. But it's still error ^_^

Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:954)
    ...
14/07/17 19:37:25 INFO mapred.JobClient:  map 0% reduce 0%
14/07/17 19:37:26 INFO mapred.JobClient: Job complete: job_local1818916530_0001
14/07/17 19:37:26 INFO mapred.JobClient: Counters: 0
14/07/17 19:37:26 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
    ...

(details log at: https://docs.google.com/document/d/1E... )

First, i set flavor for a node group template is 1GB Ram and 10GB Disk; use 1 node master and 3 node worker. Then i set 2GB Ram and 20GB Disk. But still this error :( I think the problem here is not due to lack of memory ...(more)

datts gravatar imagedatts ( 2014-07-17 15:18:40 -0500 )edit
0

answered 2014-07-21 02:51:57 -0500

datts gravatar image

updated 2014-07-21 03:01:38 -0500

Thanks a lot everyone,

I have fixed hadoop-problem :)

edit flag offensive delete link more

Comments

I am facing the same problem my machine has 1 core, 2 GB RAM and 10 GB space. Could you please share how did you resolve it? Thanks in advance!

exergy gravatar imageexergy ( 2015-01-22 06:40:47 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-07-14 11:10:37 -0500

Seen: 769 times

Last updated: Jul 21 '14