Ask Your Question
1

Storm as Openstack Data Processing

asked 2014-12-09 22:52:41 -0600

Nastooh gravatar image

updated 2014-12-29 20:05:25 -0600

Wondering what are the preparations that are needed to add Storm as a "Data Processing" plugin, to sahara.conf? Any suggestion or reference work is welcome.

Edit 1: (As per elmiko instructions.) Installed the latest Sahara:

(sahara-venv)root@cloudctrl1:~# sahara-all --config-file sahara-venv/etc/sahara.conf --version
2015.1

and updated plugins line to:

[DEFAULT]
# List of plugins to be loaded. Sahara preserves the order of
# the list when returning it. (list value)
plugins=vanilla,hdp,cdh,storm,spark

However, am getting the following error:

(sahara-venv)root@cloudctrl1:~# sahara-all --config-file sahara-venv/etc/sahara.conf --debug
2014-12-30 01:41:42.867 26233 INFO sahara.main [-] Starting Sahara all-in-one
2014-12-30 01:41:42.867 26233 INFO sahara.utils.rpc [-] Notifications disabled
2014-12-30 01:41:43.194 26233 INFO sahara.plugins.base [-] Plugin 'hdp' loaded sahara.plugins.hdp.ambariplugin:AmbariPlugin
2014-12-30 01:41:43.194 26233 INFO sahara.plugins.base [-] Plugin 'vanilla' loaded sahara.plugins.vanilla.plugin:VanillaProvider
2014-12-30 01:41:43.195 26233 INFO sahara.plugins.base [-] Plugin 'cdh' loaded sahara.plugins.cdh.plugin:CDHPluginProvider
2014-12-30 01:41:43.195 26233 INFO sahara.plugins.base [-] Plugin 'spark' loaded sahara.plugins.spark.plugin:SparkProvider
2014-12-30 01:41:43.196 26233 CRITICAL sahara [-] ConfigurationError: Plugins couldn't be loaded: storm
Error ID: 967b3625-1c55-4d60-9647-fce7eb41da1d
2014-12-30 01:41:43.196 26233 TRACE sahara Traceback (most recent call last):
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/bin/sahara-all", line 10, in <module>
2014-12-30 01:41:43.196 26233 TRACE sahara     sys.exit(main())
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/local/lib/python2.7/site-packages/sahara/cli/sahara_all.py", line 54, in main
2014-12-30 01:41:43.196 26233 TRACE sahara     server.setup_common(possible_topdir, 'all-in-one')
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/local/lib/python2.7/site-packages/sahara/main.py", line 89, in setup_common
2014-12-30 01:41:43.196 26233 TRACE sahara     plugins_base.setup_plugins()
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/local/lib/python2.7/site-packages/sahara/plugins/base.py", line 142, in setup_plugins
2014-12-30 01:41:43.196 26233 TRACE sahara     PLUGINS = PluginManager()
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/local/lib/python2.7/site-packages/sahara/plugins/base.py", line 90, in __init__
2014-12-30 01:41:43.196 26233 TRACE sahara     self._load_cluster_plugins()
2014-12-30 01:41:43.196 26233 TRACE sahara   File "/home/ubuntu/sahara-venv/local/lib/python2.7/site-packages/sahara/plugins/base.py", line 115, in _load_cluster_plugins
2014-12-30 01:41:43.196 26233 TRACE sahara     ", ".join(requested_plugins - loaded_plugins))
2014-12-30 01:41:43.196 26233 TRACE sahara ConfigurationError: Plugins couldn't be loaded: storm
2014-12-30 01:41:43.196 26233 TRACE sahara Error ID: 967b3625-1c55-4d60-9647-fce7eb41da1d
2014-12-30 01:41:43.196 26233 TRACE sahara

Also, noted that there is in fact “strom” directory under sahara/plugins. Any thoughts?

edit retag flag offensive close merge delete
add a comment

1 answer

Sort by » oldest newest most voted
1

answered 2014-12-10 09:05:51 -0600

elmiko gravatar image

Storm support has recently been added to Sahara, to use it you will need to obtain a fairly recent(Storm was merged on Nov. 27) copy of Sahara. after install you will need to enable the Storm plugin by adding it to the plugin list in your configuration file. finally, you will need to ensure that the images you are using with Sahara have Storm installed. i don't think there is a Storm image hosted online currently, so you will need to build one using the sahara-image-elements tool which can be found at https://github.com/openstack/sahara-image-elements (https://github.com/openstack/sahara-i...) .

here is a reference to the review for the Storm integration, https://review.openstack.org/#/c/137699/

edit flag offensive delete link more

Comments

Excellent news, thank you. Would “pip install sahara –upgrade” do the trick? (Do you happen to know if there would be conflict if an image contained both Spark and Strom?) Cheers,

Nastooh gravatar imageNastooh ( 2014-12-10 10:40:17 -0600 )edit

I don't think the Storm change has propagated to pypi yet, perhaps after the Kilo-1 release on Dec. 18.

As for Storm and Spark in the same image; i don't think they would conflict but you would need to manually construct the image. i don't think sahara-image-elements will create this type of image.

elmiko gravatar imageelmiko ( 2014-12-11 08:00:13 -0600 )edit

hey navesta, I worked on this integration, if you have any question on how to use it feel free to ping at #openstack-sahara, my nick is tellesnobrega or tellesnobrega_

tellesmvn gravatar imagetellesmvn ( 2014-12-11 12:10:15 -0600 )edit

Sorry for the late reply. Still having difficulties, kindly, see Edit 1, in the original post.

Nastooh gravatar imageNastooh ( 2014-12-29 20:04:23 -0600 )edit

@navesta, looks like you've run into a bug in Sahara. the problem stems from the Storm entry point not being present in the setuptools configuration for the install. i've created a bug(https://bugs.launchpad.net/sahara/+bug/1407120) to address this, thanks!

elmiko gravatar imageelmiko ( 2015-01-02 10:54:57 -0600 )edit
see more comments

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2014-12-09 22:52:41 -0600

Seen: 325 times

Last updated: Dec 29 '14