Ask Your Question
0

savanna 0.3 excuting pig script encounter error

asked 2014-02-28 06:37:28 -0500

gzsgwsgwl gravatar image

updated 2014-03-03 17:50:59 -0500

smaffulli gravatar image
script:
    A = load '$INPUT'  as (line);
    B = foreach A generate flatten(TOKENIZE(line)) as word; 
   C = group B by word;
   D = foreach C generate group, COUNT(B);
   store D into '$OUTPUT';

there is no any exceptions ocurr in jobtacker,namenode,tasktracker or datanode logs. the only exceptions which i can find in /opt/oozie/logs/oozie.log is as follows:

2014-02-28 06:12:49,347  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =  2014-02-28T06:11Z
2014-02-28 06:12:49,350  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:13:09,664  INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:13:09,687  INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:13:49,351  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:13:49,352  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =  2014-02-28T06:12Z
2014-02-28 06:13:49,355  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =  2014-02-28T06:12Z
2014-02-28 06:13:49,358  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:14:09,689  INFO PauseTransitService:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:14:09,712  INFO PauseTransitService:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2014-02-28 06:14:49,358  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:14:49,359  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running coordinator status service from last instance time =  2014-02-28T06:13Z
2014-02-28 06:14:49,362  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Running bundle status service from last instance time =  2014-02-28T06:13Z
2014-02-28 06:14:49,366  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2014-02-28 06:15:05,358  WARN LiteWorkflowAppService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://v-ndhoo-v-nn-001:8020/user/hadoop/worldcount/4496286e-d5e0-42fe-8842-800bace75f54/lib] does not exist
2014-02-28 06:15:05,499  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] Start action [0000004-140227105121167-oozie-hado-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 06:15:05,499  WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] [***0000004-140227105121167-oozie-hado-W@:start:***]Action status=DONE
2014-02-28 06:15:05,500  WARN ActionStartXCommand:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@:start:] [***0000004-140227105121167-oozie-hado-W@:start:***]Action updated in DB!
2014-02-28 06:15:05,602  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] Start action [0000004-140227105121167-oozie-hado-W@job-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2014-02-28 06:15:06,029  WARN PigActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[job-wf] JOB[0000004-140227105121167-oozie-hado-W] ACTION[0000004-140227105121167-oozie-hado-W@job-node] credentials is null for the action
2014-02-28 06:15:06 ...
(more)
edit retag flag offensive close merge delete

3 answers

Sort by » oldest newest most voted
1

answered 2014-02-28 15:43:14 -0500

alazarev gravatar image

I see the following line in your logs

2014-02-28 06:15:05,358 WARN LiteWorkflowAppService:542 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://v-ndhoo-v-nn-001:8020/user/hadoop/worldcount/4496286e-d5e0-42fe-8842-800bace75f54/lib] does not exist

'lib' folder usually contains all libraries attached to the job. Most probably you forgot to add .jar file with classes you use in pig script (second tab on UI during Job creation).

edit flag offensive delete link more
0

answered 2014-03-05 01:53:56 -0500

gws gravatar image

find the reason which occurred mainly because of my wrong swift configuration。 after i apply savanna's 1240144 and 1240511 two bugs, then change swift://demo/input to swift://demo.savanna/input, and every things start to work well.

i am very sorry to bother you all just because my carefulless. thank all of you !

edit flag offensive delete link more
0

answered 2014-03-06 21:52:11 -0500

soloman_hao gravatar image

hi,can u help to find what's wrong with my deployment,logs in /var/log/savanna/api.log are below when launch a job

2014-03-07 19:05:47.560 29251 DEBUG savanna.utils.api [-] Rest.route.decorator.handler, kwargs={'tenant_id': u'f62963ebea08405590c742655c3a0d86', 'job_execution_id': u'8e9d06f5-b4b3-4931-b431-da7d8810e2a2'} handler /usr/lib/python2.6/site-packages/savanna/utils/api.py:63

2014-03-07 19:06:12.232 29251 ERROR savanna.service.edp.job_manager [-] Error during update job execution 765538cc-39e4-440a-a937-5f90311cad69: [Errno 110] ETIMEDOUT

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager Traceback (most recent call last):

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/savanna/service/edp/job_manager.py", line 74, in update_job_statuses

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager get_job_status(je.id)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/savanna/service/edp/job_manager.py", line 60, in get_job_status

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager job_info = client.get_job_status(job_execution.oozie_job_id)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/savanna/service/edp/oozie.py", line 45, in get_job_status

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager resp = requests.get(self.job_url % job_id + "?show=info")

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager return request('get', url, **kwargs)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site- packages/requests/api.py", line 44, in request 2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager return session.request(method=method, url=url, **kwargs) 2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager r = adapter.send(request, **kwargs)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 206, in send

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager raise ConnectionError(sockerr)

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager ConnectionError: [Errno 110] ETIMEDOUT

2014-03-07 19:06:12.232 29251 TRACE savanna.service.edp.job_manager

2014-03-07 19:06:12.234 29251 DEBUG savanna.openstack.common.periodic_task [-] Running periodic task SavannaPeriodicTasks.terminate_unneeded_clusters run_periodic_tasks /usr/lib/python2.6/site-packages/savanna/openstack/common/periodic_task.py:177

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-02-28 06:37:28 -0500

Seen: 370 times

Last updated: Mar 06 '14