Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Juno Sahara Spark 1.0.0 Security Group Error

Hi
Having problem with security groups, when either defining "Node Group Templates" or instantiating a cluster. In the first case, if I use an existing group, say "default", gui shows an error stating "Error Security group '2' not found". Sahara log file indicates the same thing:

   2014-12-03 23:41:24.144 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
    2014-12-03 23:41:24.146 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
    2014-12-03 23:41:24.163 30234 DEBUG urllib3.connectionpool [-] "GET /v2/926c31c887f441f6a4e4b8031b8cc528/os-security-groups HTTP/1.1" 200 682 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
    2014-12-03 23:41:24.165 30234 DEBUG sahara.utils.api [-] Validation Error occurred: error_code=400, error_message=Security group '2' not found, error_name=INVALID_REFERENCE bad_request /usr/local/lib/python2.7/dist-packages/sahara/utils/api.py:245
    2014-12-03 23:41:24.165 30234 INFO sahara.cli.sahara_all [-] 10.0.0.86 - - [03/Dec/2014 23:41:24] "POST /v1.1/926c31c887f441f6a4e4b8031b8cc528/node-group-templates HTTP/1.1" 400 221 0.063121
    2014-12-03 23:41:24.257 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
    2014-12-03 23:41:24.258 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

I can, temporarily, avoid the problem by selecting "Auto Security Group" option. This would allow for a node group to be created; however, I do not see any new security group, under Compute -> Access & Security. At any rate, this also fails during cluster instantiation:

2014-12-03 23:55:06.285 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
2014-12-03 23:55:06.286 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
2014-12-03 23:55:06.409 30234 DEBUG urllib3.connectionpool [-] "POST /v2/926c31c887f441f6a4e4b8031b8cc528/servers HTTP/1.1" 400 116 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-12-03 23:55:06.451 30234 ERROR sahara.service.ops [-] Error during operating cluster 'Sprk265' (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops Traceback (most recent call last):
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 113, in wrapper
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     f(cluster_id, *args, **kwds)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 198, in _provision_cluster
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     INFRA.create_cluster(cluster)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 51, in create_cluster
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._create_instances(cluster)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 168, in _create_instances
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._run_instance(cluster, node_group, idx, aa_group=aa_group)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 305, in _run_instance
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **nova_kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 883, in create
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **boot_kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 546, in _boot
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return_raw=return_raw, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 100, in _create
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     _resp, body = self.api.client.post(url, body=body)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 490, in post
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return self._cs_request(url, 'POST', **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 465, in _cs_request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self._time_request(url, method, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in _time_request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self.request(url, method, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 433, in request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     raise exceptions.from_response(resp, body, url, method)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops BadRequest: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops 
2014-12-03 23:55:06.611 30234 INFO sahara.service.direct_engine [-] Cluster 'Sprk265' creation rollback (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
2014-12-03 23:55:06.616 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
2014-12-03 23:55:06.622 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
2014-12-03 23:55:07.018 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
2014-12-03 23:55:07.019 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

Here is my conf file:

[DEFAULT]
use_floating_ips=True
use_neutron=True
 [keystone_authtoken]
auth_uri = http://keystone1.maas17:5000/v2.0/
identity_uri=http://keystone1.maas17:35357/
admin_user=sahara
admin_password=sahara
admin_tenant_name=sahara
periodic_enable=true
plugins=vanilla,hdp,idh,spark,cdh
[database]
connection=mysql://sahara:sahara@mysql1.maas17/sahara

Sahara is also registered with Keystone:

# keystone service-list
+----------------------------------+----------+-----------------+----------------------------+
|                id                |   name   |       type      |        description         |
+----------------------------------+----------+-----------------+----------------------------+
| 75b2c466c35a44d5bbe7167c1ed38e20 |  cinder  |      volume     |   Cinder Volume Service    |
| d5b014c2d96e4d619f9d9b8e646f0f5b |   ec2    |       ec2       |  EC2 Compatibility Layer   |
| 8180061e79a24627be43485910a9e16a |  glance  |      image      |    Glance Image Service    |
| 4dad7b5145c842a4a8fbbab1f158629c | keystone |     identity    | Keystone Identity Service  |
| d833594550e14d49be4df4394886c849 |   nova   |     compute     |    Nova Compute Service    |
| fe42b5328002433989ffd6b18414aacc | quantum  |     network     | Quantum Networking Service |
| fac3f901841c4528b3902ae1b7265b4e |    s3    |        s3       | S3 Compatible object-store |
| 12104e23db1441a28cb42cf8c9437139 |  sahara  | data_processing |  Data processing service   |
+----------------------------------+----------+-----------------+----------------------------+

Any thoughts?

Juno Sahara Spark 1.0.0 Security Group Error

Hi
Having problem with security groups, when either defining "Node Group Templates" or instantiating a cluster. In the first case, if I use an existing group, say "default", gui shows an error stating "Error Security group '2' not found". Sahara log file indicates the same thing:

   2014-12-03 23:41:24.144 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
    2014-12-03 23:41:24.146 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
    2014-12-03 23:41:24.163 30234 DEBUG urllib3.connectionpool [-] "GET /v2/926c31c887f441f6a4e4b8031b8cc528/os-security-groups HTTP/1.1" 200 682 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
    2014-12-03 23:41:24.165 30234 DEBUG sahara.utils.api [-] Validation Error occurred: error_code=400, error_message=Security group '2' not found, error_name=INVALID_REFERENCE bad_request /usr/local/lib/python2.7/dist-packages/sahara/utils/api.py:245
    2014-12-03 23:41:24.165 30234 INFO sahara.cli.sahara_all [-] 10.0.0.86 - - [03/Dec/2014 23:41:24] "POST /v1.1/926c31c887f441f6a4e4b8031b8cc528/node-group-templates HTTP/1.1" 400 221 0.063121
    2014-12-03 23:41:24.257 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
    2014-12-03 23:41:24.258 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

I can, temporarily, avoid the problem by selecting "Auto Security Group" option. This would allow for a node group to be created; however, I do not see any new security group, under Compute -> Access & Security. At any rate, this also fails during cluster instantiation:

2014-12-03 23:55:06.285 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
2014-12-03 23:55:06.286 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
2014-12-03 23:55:06.409 30234 DEBUG urllib3.connectionpool [-] "POST /v2/926c31c887f441f6a4e4b8031b8cc528/servers HTTP/1.1" 400 116 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-12-03 23:55:06.451 30234 ERROR sahara.service.ops [-] Error during operating cluster 'Sprk265' (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops Traceback (most recent call last):
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 113, in wrapper
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     f(cluster_id, *args, **kwds)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/ops.py", line 198, in _provision_cluster
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     INFRA.create_cluster(cluster)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 51, in create_cluster
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._create_instances(cluster)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 168, in _create_instances
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     self._run_instance(cluster, node_group, idx, aa_group=aa_group)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/local/lib/python2.7/dist-packages/sahara/service/direct_engine.py", line 305, in _run_instance
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **nova_kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 883, in create
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     **boot_kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 546, in _boot
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return_raw=return_raw, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 100, in _create
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     _resp, body = self.api.client.post(url, body=body)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 490, in post
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     return self._cs_request(url, 'POST', **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 465, in _cs_request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self._time_request(url, method, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in _time_request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     resp, body = self.request(url, method, **kwargs)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 433, in request
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops     raise exceptions.from_response(resp, body, url, method)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops BadRequest: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400)
2014-12-03 23:55:06.451 30234 TRACE sahara.service.ops 
2014-12-03 23:55:06.611 30234 INFO sahara.service.direct_engine [-] Cluster 'Sprk265' creation rollback (reason: Security group 6 not found for project 926c31c887f441f6a4e4b8031b8cc528. (HTTP 400))
2014-12-03 23:55:06.616 30234 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): cloudctrl1.maas17
2014-12-03 23:55:06.622 30234 DEBUG urllib3.connectionpool [-] Setting read timeout to None _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:375
2014-12-03 23:55:07.018 30234 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:650
2014-12-03 23:55:07.019 30234 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py:707

Here is my conf file:

[DEFAULT]
use_floating_ips=True
use_neutron=True
 [keystone_authtoken]
auth_uri = http://keystone1.maas17:5000/v2.0/
identity_uri=http://keystone1.maas17:35357/
admin_user=sahara
admin_password=sahara
admin_tenant_name=sahara
periodic_enable=true
plugins=vanilla,hdp,idh,spark,cdh
[database]
connection=mysql://sahara:sahara@mysql1.maas17/sahara

Sahara is also registered with Keystone:

# keystone service-list
+----------------------------------+----------+-----------------+----------------------------+
|                id                |   name   |       type      |        description         |
+----------------------------------+----------+-----------------+----------------------------+
| 75b2c466c35a44d5bbe7167c1ed38e20 |  cinder  |      volume     |   Cinder Volume Service    |
| d5b014c2d96e4d619f9d9b8e646f0f5b |   ec2    |       ec2       |  EC2 Compatibility Layer   |
| 8180061e79a24627be43485910a9e16a |  glance  |      image      |    Glance Image Service    |
| 4dad7b5145c842a4a8fbbab1f158629c | keystone |     identity    | Keystone Identity Service  |
| d833594550e14d49be4df4394886c849 |   nova   |     compute     |    Nova Compute Service    |
| fe42b5328002433989ffd6b18414aacc | quantum  |     network     | Quantum Networking Service |
| fac3f901841c4528b3902ae1b7265b4e |    s3    |        s3       | S3 Compatible object-store |
| 12104e23db1441a28cb42cf8c9437139 |  sahara  | data_processing |  Data processing service   |
+----------------------------------+----------+-----------------+----------------------------+

Any thoughts?

Edit 1: I can get further by avoiding the selection of Security Group, altogther. In fact, I can launch a cluster now! (Having problem with login into the image, but that is a different post!)