the username and password for swift is gone when trying the java type job in sahara
Dear stackers,
We are figuring out how to use the job type job in sahara in master branch by devstack. we tried 3 ways:
- using horizon
- using sahara client
- using rest API
However non of above method can passing the username and password for swift into hadoop cluster. Is this a potential issue for current build or not?
example to use sahara API to create job type job
request body:-- job.json
{
"job_execution": {
"id": "ae97c393-073d-48be-aa4f-c4686304a4cf",
"output_id": null,
"info": {
"status": "PENDING"
},
"job_id": "9b6deedf-740a-4d9d-a2fd-95d2c204318b",
"tenant_id": "c42d79bf307a471aa771eb786593379e",
"created_at": "2014-08-27 07:41:06.457561",
"input_id": null,
"cluster_id": "06d25d10-5daf-42f4-8f32-788131244064",
"job_configs": {
"params": {},
"args": [
"wordcount",
"swift://input.sahara/config-reference-icehouse",
"swift://result.sahara/output1"
],
"configs": {
"fs.swift.service.sahara.username": "",
"mapred.reduce.tasks": "5",
"fs.swift.service.sahara.password": "",
"mapred.map.tasks": "10",
"edp.java.main_class": "org.apache.hadoop.examples.ExampleDriver"
}
}
}
}
JOB_ID=$(curl -X POST -d @job.json -H "X-Auth-Token:$TOKEN" -H "Content-Type: application/json" http://192.85.180.200:8386/v1.1/c42d79bf307a471aa771eb786593379e/jobs/9b6deedf-740a-4d9d-a2fd-95d2c204318b/execute)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1200 100 681 100 519 5326 4059 --:--:-- --:--:-- --:--:-- 5362
stack@sahara:~$ echo $JOB_ID
{"job_execution": {"output_id": null, "info": {"status": "PENDING"}, "job_id": "9b6deedf-740a-4d9d-a2fd-95d2c204318b", "tenant_id": "c42d79bf307a471aa771eb786593379e", "created_at": "2014-08-27 07:41:06.457561", "input_id": null, "cluster_id": "06d25d10-5daf-42f4-8f32-788131244064", "job_configs": {"configs": {"edp.java.main_class": "org.apache.hadoop.examples.ExampleDriver", "mapred.map.tasks": "10", **"fs.swift.service.sahara.password": ""**, "mapred.reduce.tasks": "5", "**fs.swift.service.sahara.username": ""**}, "args": ["wordcount", "swift://input.sahara/config-reference-icehouse", "swift://result.sahara/output1"], "params": {}}, "id": "ae97c393-073d-48be-aa4f-c4686304a4cf"}}
in the above response we can see the fs.swift.service.sahara.username and fs.swift.service.sahara.password values turned into "" this will cause the hadoop complains the following error message:
org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Missing mandatory configuration option: fs.swift.service.sahara.username
at org.apache.hadoop.fs.swift.http.RestClientBindings.copy(RestClientBindings.java:217)
at org.apache.hadoop.fs.swift.http.RestClientBindings.bind(RestClientBindings.java:150)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.<init>(SwiftRestClient.java:467)
at org.apache.hadoop.fs.swift.http.SwiftRestClient.getInstance(SwiftRestClient.java:1599)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.initialize(SwiftNativeFileSystemStore.java:81)
at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.initialize(SwiftNativeFileSystem.java:125)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
add a comment
