Ask Your Question
0

Does Savanna have local storage?

asked 2013-09-12 10:19:42 -0500

I've just watched a presentation on youtube and I'm unexperimented with open-stack.

What are the options to manage the data layer? I've seen there is a plugin for switf but how can I have my data on the same physical host as the dn/tt to limit the network I/O?

Thanks, Guillaume PS: Is there a mailing-list ?

edit retag flag offensive close merge delete

5 answers

Sort by ยป oldest newest most voted
0

answered 2013-09-12 15:06:54 -0500

Thanks Alexander Kuznetsov, that solved my question.

edit flag offensive delete link more
0

answered 2013-09-12 14:46:11 -0500

To provide a good performance for Hadoop cluster in a virtual environment, Savanna has on the road map feature a directly attached disc for vms. The patch for this already committed to Cinder and Savanna will support this functionality in the future. Also, you can look at VMWare paper http://www.mellanox.com/pdf/case_studies/VMW-Hadoop-Performance-vSphere5.pdf (http://www.mellanox.com/pdf/case_stud...) about Hadoop performance in the virtual environment.

edit flag offensive delete link more
0

answered 2013-09-12 12:37:21 -0500

Hi Alexander,

I'm working on a traditionnal hadoop cluster with only physical hosts.

In the case of a production hadoop cluster (or long-lived cluster), I want the best performance. For that, I need to limit all bottlenecks especially the network I/O. So I need to have a local storage for the worker node.

Is it possible to have a kind of that architecture if my cluster is hosted on open-stack. How can I create a virtual worker-node which can read the input data from the hosts's physical disks for instance.

If I use swift for the input/output data, data has to read/write through the network, is that right? I don't want that in my case.

edit flag offensive delete link more
0

answered 2013-09-12 10:39:53 -0500

mattf gravatar image

Be sure to put [savanna] in the subject of your message to openstack-dev@...

edit flag offensive delete link more
0

answered 2013-09-12 10:31:51 -0500

Hi, Guillaume,

Can you explain your usecase in details. As I understand from your question you wants to run M/R jobs over your own data, right? In that case you need to copy your data to provisioned cluster by Savanna, or copy data to Swift which is suppoted by Hadoop provisioned by Savanna.

The proper email address: openstack-dev@lists.openstack.org

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-09-12 10:19:42 -0500

Seen: 11 times

Last updated: Sep 12 '13