Spark Kilo Nova Fails to Deploy:

asked 2015-06-12 19:23:00 -0500

Nastooh gravatar image

Hi
Trying to deploy a 4-node spark cluster, 1 master and 3 slaves, and am running into scheduler problem:

    # tail -f /var/log/nova/nova-conductor.log
    2015-06-13 00:13:14.119 1030 INFO oslo_messaging._drivers.impl_rabbit [req-62362f47-167e-4a23-8c9b-c50d94448cbf fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] Connecting to AMQP server on 10.4.0.60:5672
2015-06-13 00:13:14.140 1030 INFO oslo_messaging._drivers.impl_rabbit [req-62362f47-167e-4a23-8c9b-c50d94448cbf fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] Connected to AMQP server on 10.4.0.60:5672
2015-06-13 00:13:20.750 1025 INFO oslo_messaging._drivers.impl_rabbit [req-27b42398-5b52-4762-abef-3baa75624b3e fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] Connecting to AMQP server on 10.4.0.60:5672
2015-06-13 00:13:20.763 1025 INFO oslo_messaging._drivers.impl_rabbit [req-27b42398-5b52-4762-abef-3baa75624b3e fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] Connected to AMQP server on 10.4.0.60:5672
2015-06-13 00:13:20.793 1025 WARNING nova.scheduler.utils [req-27b42398-5b52-4762-abef-3baa75624b3e fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 86, in select_destinations
    filter_properties)

  File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 80, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

2015-06-13 00:13:20.794 1025 WARNING nova.scheduler.utils [req-27b42398-5b52-4762-abef-3baa75624b3e fcbce29fa1eb4b42921a20055afdc024 8f3633c4ba1640abbab3be608b6bcc88 - - -] [instance: c752241f-f07d-46cd-8844-f1747575e8a2] Setting instance to ERROR state.

And sahara log:

# tail -f /var/log/sahara.log  -n 350 | more
  2015-06-13 00:12:57.774 1968 DEBUG keystonemiddleware.auth_token [-] Removing headers from request environment: X-Service-Catalog,X-Identity-Status,X-Service-Identity-Status,X-Roles,X-Service-Roles,X-Domain-Name,X-Service-Domain-Name,X-Project-Id,X-Service-Project-Id,X-Project-Domain-Name,X-Service-Project-Domain-Name,X-User-Id,X-Service-User-Id,X-User-Name,X-Service-User-Name,X-Project-Name,X-Service-Project-Name,X-User-Domain-Id,X-Service-User-Domain-Id,X-Domain-Id,X-Service-Domain-Id,X-User-Domain-Name,X-Service-User-Domain-Name,X-Project-Domain-Id,X-Service-Project-Domain-Id,X-Role,X-User,X-Tenant-Name,X-Tenant-Id,X-Tenant _remove_auth_headers /opt/sahara/sahara-venv-stable-kilo/local/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:672
2015-06-13 00:12:57.775 1968 DEBUG keystonemiddleware.auth_token [-] Authenticating user token __call__ /opt/sahara/sahara-venv-stable-kilo/local/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:581
2015-06-13 00:12:57.776 1968 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://10.4.0.61:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}9774f30871278b81c54e43e1d87e51915f38a60d" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}964e5e4bfa51abcd0c3f254409db775b35e3ce9c" _http_log_request /opt/sahara/sahara-venv-stable-kilo/local/lib/python2.7/site-packages/keystoneclient/session.py:195
2015-06-13 00:12:57.940 1968 DEBUG keystoneclient.session [-] RESP: [200] content-length: 6583 x-subject-token: {SHA1}9774f30871278b81c54e43e1d87e51915f38a60d vary: X-Auth-Token x-distribution: Ubuntu connection: keep-alive date: Sat, 13 Jun 2015 00:12:57 GMT content-type: application/json x-openstack-request-id: req-f2857800-c607-4410-ba2a-c8ffc97b5351 
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}, {"id": "f5c52968f8af4111bbfccd39ef500edd", "name": "Admin"}], "expires_at": "2015-06-13T01:12:57.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "8f3633c4ba1640abbab3be608b6bcc88", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "fcbce29fa1eb4b42921a20055afdc024", "name": "admin"}, "audit_ids": ["6jslb2MQS-2ySURQMeLvDw"], "issued_at": "2015-06-13T00:12:57.643005"}}
 _http_log_response /opt/sahara/sahara-venv-stable-kilo/local/lib/python2.7/site-packages/keystoneclient/session.py:224
2015-06-13 00:12:57.941 1968 DEBUG keystonemiddleware.auth_token [-] Storing token in cache store /opt/sahara/sahara-venv-stable-kilo/local/lib ...
(more)
edit retag flag offensive close merge delete

Comments

Problem was due to anti-affinity definition that include both master and slave nodes, which in turn would've required 4 separate compute nodes, all together.

Nastooh gravatar imageNastooh ( 2015-06-15 17:36:44 -0500 )edit