Ask Your Question
0

the username and password for swift is gone when trying the java type job in sahara

asked 2014-08-27 08:05:22 -0600

9lives gravatar image

Dear stackers,

We are figuring out how to use the job type job in sahara in master branch by devstack. we tried 3 ways:

  1. using horizon
  2. using sahara client
  3. using rest API

However non of above method can passing the username and password for swift into hadoop cluster. Is this a potential issue for current build or not?

example to use sahara API to create job type job

request body:-- job.json

{
  "job_execution": {
    "id": "ae97c393-073d-48be-aa4f-c4686304a4cf",
    "output_id": null,
    "info": {
      "status": "PENDING"
    },
    "job_id": "9b6deedf-740a-4d9d-a2fd-95d2c204318b",
    "tenant_id": "c42d79bf307a471aa771eb786593379e",
    "created_at": "2014-08-27 07:41:06.457561",
    "input_id": null,
    "cluster_id": "06d25d10-5daf-42f4-8f32-788131244064",
    "job_configs": {
      "params": {},
      "args": [
        "wordcount",
        "swift://input.sahara/config-reference-icehouse",
        "swift://result.sahara/output1"
      ],
      "configs": {
        "fs.swift.service.sahara.username": "",
        "mapred.reduce.tasks": "5",
        "fs.swift.service.sahara.password": "",
        "mapred.map.tasks": "10",
        "edp.java.main_class": "org.apache.hadoop.examples.ExampleDriver"
      }
    }
  }
}
JOB_ID=$(curl -X POST   -d @job.json -H "X-Auth-Token:$TOKEN"   -H "Content-Type: application/json" http://192.85.180.200:8386/v1.1/c42d79bf307a471aa771eb786593379e/jobs/9b6deedf-740a-4d9d-a2fd-95d2c204318b/execute)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1200  100   681  100   519   5326   4059 --:--:-- --:--:-- --:--:--  5362
stack@sahara:~$ echo $JOB_ID
{"job_execution": {"output_id": null, "info": {"status": "PENDING"}, "job_id": "9b6deedf-740a-4d9d-a2fd-95d2c204318b", "tenant_id": "c42d79bf307a471aa771eb786593379e", "created_at": "2014-08-27 07:41:06.457561", "input_id": null, "cluster_id": "06d25d10-5daf-42f4-8f32-788131244064", "job_configs": {"configs": {"edp.java.main_class": "org.apache.hadoop.examples.ExampleDriver", "mapred.map.tasks": "10", **"fs.swift.service.sahara.password": ""**, "mapred.reduce.tasks": "5", "**fs.swift.service.sahara.username": ""**}, "args": ["wordcount", "swift://input.sahara/config-reference-icehouse", "swift://result.sahara/output1"], "params": {}}, "id": "ae97c393-073d-48be-aa4f-c4686304a4cf"}}

in the above response we can see the fs.swift.service.sahara.username and fs.swift.service.sahara.password values turned into "" this will cause the hadoop complains the following error message:

org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username
    at org.apache.hadoop.fs.swift.http.RestClientBindings.copy(RestClientBindings.java:217)
    at org.apache.hadoop.fs.swift.http.RestClientBindings.bind(RestClientBindings.java:150)
    at org.apache.hadoop.fs.swift.http.SwiftRestClient.<init>(SwiftRestClient.java:467)
    at org.apache.hadoop.fs.swift.http.SwiftRestClient.getInstance(SwiftRestClient.java:1599)
    at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.initialize(SwiftNativeFileSystemStore.java:81)
    at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.initialize(SwiftNativeFileSystem.java:125)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2014-09-09 16:09:28 -0600

alazarev gravatar image

Sahara sanitizes swift username/password for security reasons. That's why they are empty in responses. This has nothing to do with hadoop error.

Could you please provide generated oozie workflow file? It is usually located in HDFS under /user/<user_name>/<job_name>

edit flag offensive delete link more

Comments

Hi alazarev ,

Thanks for the answer, here is the workflow generated by oozie http://paste.openstack.org/show/112373/

9lives gravatar image9lives ( 2014-09-16 21:56:03 -0600 )edit

I see "fs.swift.service.sahara.username" option there.

alazarev gravatar imagealazarev ( 2014-09-16 23:14:02 -0600 )edit

what does this option mean? Do you mean this option should not be existed in workflow.xml? could you give a detailed example for this?

Thanks!

9lives gravatar image9lives ( 2014-09-18 01:14:18 -0600 )edit

@9lives, @alazarev I got exactly the same problem. Did you manage to resolve it?

radek gravatar imageradek ( 2014-12-13 07:01:38 -0600 )edit
0

answered 2014-12-14 04:16:12 -0600

radek gravatar image

updated 2014-12-14 04:20:32 -0600

I got the same problem and it turned out that I simply did not read thouroughly the documentation at https://github.com/openstack/sahara/blob/master/etc/edp-examples/edp-java/README.rst (https://github.com/openstack/sahara/b...) . To run a java job on sahara you MUST alter its source file so that it contains the following line:

conf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml")));

This line causes that everything that is under <configuration> tag in an Oozie workflow is passed to the Configuration object, especially fs.swift.service.sahara.username and fs.swift.service.sahara.password values.

You can always check whether they were set by going to oozie (you'll find the link on the _General Info_ page of your cluster in Horizon), clicking on a job and then "Job Definition".

My hadoop version is 2.3.0, open stack: icehouse.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-08-27 08:05:22 -0600

Seen: 720 times

Last updated: Dec 14 '14