Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up multi-domain support, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:

DEBUG = True
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea is that the UI should know what different users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up multi-domain support, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:

DEBUG = True
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea is that the UI should know what different users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.


In response to mpetason, here is the service list:

# keystone service-list
+----------------------------------+------------+--------------+--------------------------------+
|                id                |    name    |     type     |          description           |
+----------------------------------+------------+--------------+--------------------------------+
| b289d7f1545a49d2a35e64db07ab9c3b | ceilometer |   metering   |   Openstack Metering Service   |
| 2273ffaebf3949bc8460365e78e190d0 |   cinder   |    volume    |         Cinder Service         |
| 08cb46b56b8546beb62bc8f2e1426ed9 | cinder_v2  |   volumev2   |       Cinder Service v2        |
| e5d03c64d2b14b9db55c0ce60f9debca |  cinderv2  |   volumev2   |       Cinder Service v2        |
| df971f50591c40ff974aa535a2c9138c |   glance   |    image     |    Openstack Image Service     |
| c95586316d1f41ab9af05dd7efd09bbb |  keystone  |   identity   |   OpenStack Identity Service   |
| 57ec26fda6ae4596b88a42c0a32a5d78 |  neutron   |   network    |   Neutron Networking Service   |
| 23ff0372d10c4396831a22c5e0a25f93 |    nova    |   compute    |   Openstack Compute Service    |
| cd32c1f660b349cdaa16ef1ff9a2bd54 |  nova_ec2  |     ec2      |          EC2 Service           |
| c7c107af6d014e69955f9b0b9d7ec0e6 |   swift    | object-store | Openstack Object-Store Service |
| 1b7f416c5a2d48e5a96728e210bcb770 |  swift_s3  |      s3      |      Openstack S3 Service      |
+----------------------------------+------------+--------------+--------------------------------+

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up multi-domain support, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:

DEBUG = True
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea is that the UI should know what different users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.


In response to mpetason, here is the service list:

# keystone service-list
+----------------------------------+------------+--------------+--------------------------------+
|                id                |    name    |     type     |          description           |
+----------------------------------+------------+--------------+--------------------------------+
| b289d7f1545a49d2a35e64db07ab9c3b | ceilometer |   metering   |   Openstack Metering Service   |
| 2273ffaebf3949bc8460365e78e190d0 |   cinder   |    volume    |         Cinder Service         |
| 08cb46b56b8546beb62bc8f2e1426ed9 | cinder_v2  |   volumev2   |       Cinder Service v2        |
| e5d03c64d2b14b9db55c0ce60f9debca |  cinderv2  |   volumev2   |       Cinder Service v2        |
| df971f50591c40ff974aa535a2c9138c |   glance   |    image     |    Openstack Image Service     |
| c95586316d1f41ab9af05dd7efd09bbb |  keystone  |   identity   |   OpenStack Identity Service   |
| 57ec26fda6ae4596b88a42c0a32a5d78 |  neutron   |   network    |   Neutron Networking Service   |
| 23ff0372d10c4396831a22c5e0a25f93 |    nova    |   compute    |   Openstack Compute Service    |
| cd32c1f660b349cdaa16ef1ff9a2bd54 |  nova_ec2  |     ec2      |          EC2 Service           |
| c7c107af6d014e69955f9b0b9d7ec0e6 |   swift    | object-store | Openstack Object-Store Service |
| 1b7f416c5a2d48e5a96728e210bcb770 |  swift_s3  |      s3      |      Openstack S3 Service      |
+----------------------------------+------------+--------------+--------------------------------+

and the endpoint-list

# keystone endpoint-list
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
|                id                |   region  |                    publicurl                     |                   internalurl                    |                   adminurl                  |            service_id            |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
| 48ac9fde57b5433a84f1dfff7c96fe04 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | e5d03c64d2b14b9db55c0ce60f9debca |
| 507a6b832402452b90aa7fb233683259 | RegionOne |         http://xxx.xxx.xxx.xx:5000/v2.0          |         http://xxx.xxx.xxx.xx:5000/v2.0          |       http://xxx.xxx.xxx.xx:35357/v2.0      | c95586316d1f41ab9af05dd7efd09bbb |
| 789c19eb25174c94945337621c2143df | RegionOne |            http://xxx.xxx.xxx.xx:8080            |            http://xxx.xxx.xxx.xx:8080            |          http://xxx.xxx.xxx.xx:8080         | 1b7f416c5a2d48e5a96728e210bcb770 |
| 7c39239bdbd94af48836a09b093c1b1e | RegionOne |            http://xxx.xxx.xxx.xx:8777            |            http://xxx.xxx.xxx.xx:8777            |          http://xxx.xxx.xxx.xx:8777         | b289d7f1545a49d2a35e64db07ab9c3b |
| 985c9cf36c784c16a54bdb8c9520c687 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | 08cb46b56b8546beb62bc8f2e1426ed9 |
| a325af2a574b41308ae7cf4e5d234f13 | RegionOne |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |  http://xxx.xxx.xxx.xx:8773/services/Admin  | cd32c1f660b349cdaa16ef1ff9a2bd54 |
| a39532546fcd4a26ab0c027c0a920269 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s | 2273ffaebf3949bc8460365e78e190d0 |
| be86120cd43245fea4c412d0c8ad6c34 | RegionOne | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s |         http://xxx.xxx.xxx.xx:8080/         | c7c107af6d014e69955f9b0b9d7ec0e6 |
| bfd29a18db84492e81c3adc5ab6bcbf3 | RegionOne |           http://xxx.xxx.xxx.xx:9696/            |           http://xxx.xxx.xxx.xx:9696/            |         http://xxx.xxx.xxx.xx:9696/         | 57ec26fda6ae4596b88a42c0a32a5d78 |
| cf75ad45a0cf4d42aea2e56c8d52c588 | RegionOne |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s | 23ff0372d10c4396831a22c5e0a25f93 |
| df8dbbe9feb546498b19cb2976daa724 | RegionOne |            http://xxx.xxx.xxx.xx:9292            |            http://xxx.xxx.xxx.xx:9292            |          http://xxx.xxx.xxx.xx:9292         | df971f50591c40ff974aa535a2c9138c |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up multi-domain support, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:

DEBUG = True
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea is that the UI should know what different users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.


In response to mpetason, here is the service list:

# keystone service-list
+----------------------------------+------------+--------------+--------------------------------+
|                id                |    name    |     type     |          description           |
+----------------------------------+------------+--------------+--------------------------------+
| b289d7f1545a49d2a35e64db07ab9c3b | ceilometer |   metering   |   Openstack Metering Service   |
| 2273ffaebf3949bc8460365e78e190d0 |   cinder   |    volume    |         Cinder Service         |
| 08cb46b56b8546beb62bc8f2e1426ed9 | cinder_v2  |   volumev2   |       Cinder Service v2        |
| e5d03c64d2b14b9db55c0ce60f9debca |  cinderv2  |   volumev2   |       Cinder Service v2        |
| df971f50591c40ff974aa535a2c9138c |   glance   |    image     |    Openstack Image Service     |
| c95586316d1f41ab9af05dd7efd09bbb |  keystone  |   identity   |   OpenStack Identity Service   |
| 57ec26fda6ae4596b88a42c0a32a5d78 |  neutron   |   network    |   Neutron Networking Service   |
| 23ff0372d10c4396831a22c5e0a25f93 |    nova    |   compute    |   Openstack Compute Service    |
| cd32c1f660b349cdaa16ef1ff9a2bd54 |  nova_ec2  |     ec2      |          EC2 Service           |
| c7c107af6d014e69955f9b0b9d7ec0e6 |   swift    | object-store | Openstack Object-Store Service |
| 1b7f416c5a2d48e5a96728e210bcb770 |  swift_s3  |      s3      |      Openstack S3 Service      |
+----------------------------------+------------+--------------+--------------------------------+

and the endpoint-list

# keystone endpoint-list
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
|                id                |   region  |                    publicurl                     |                   internalurl                    |                   adminurl                  |            service_id            |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
| 48ac9fde57b5433a84f1dfff7c96fe04 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | e5d03c64d2b14b9db55c0ce60f9debca |
| 507a6b832402452b90aa7fb233683259 | RegionOne |         http://xxx.xxx.xxx.xx:5000/v2.0          |         http://xxx.xxx.xxx.xx:5000/v2.0          |       http://xxx.xxx.xxx.xx:35357/v2.0      | c95586316d1f41ab9af05dd7efd09bbb |
| 789c19eb25174c94945337621c2143df | RegionOne |            http://xxx.xxx.xxx.xx:8080            |            http://xxx.xxx.xxx.xx:8080            |          http://xxx.xxx.xxx.xx:8080         | 1b7f416c5a2d48e5a96728e210bcb770 |
| 7c39239bdbd94af48836a09b093c1b1e | RegionOne |            http://xxx.xxx.xxx.xx:8777            |            http://xxx.xxx.xxx.xx:8777            |          http://xxx.xxx.xxx.xx:8777         | b289d7f1545a49d2a35e64db07ab9c3b |
| 985c9cf36c784c16a54bdb8c9520c687 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | 08cb46b56b8546beb62bc8f2e1426ed9 |
| a325af2a574b41308ae7cf4e5d234f13 | RegionOne |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |  http://xxx.xxx.xxx.xx:8773/services/Admin  | cd32c1f660b349cdaa16ef1ff9a2bd54 |
| a39532546fcd4a26ab0c027c0a920269 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s | 2273ffaebf3949bc8460365e78e190d0 |
| be86120cd43245fea4c412d0c8ad6c34 | RegionOne | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s |         http://xxx.xxx.xxx.xx:8080/         | c7c107af6d014e69955f9b0b9d7ec0e6 |
| bfd29a18db84492e81c3adc5ab6bcbf3 | RegionOne |           http://xxx.xxx.xxx.xx:9696/            |           http://xxx.xxx.xxx.xx:9696/            |         http://xxx.xxx.xxx.xx:9696/         | 57ec26fda6ae4596b88a42c0a32a5d78 |
| cf75ad45a0cf4d42aea2e56c8d52c588 | RegionOne |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s | 23ff0372d10c4396831a22c5e0a25f93 |
| df8dbbe9feb546498b19cb2976daa724 | RegionOne |            http://xxx.xxx.xxx.xx:9292            |            http://xxx.xxx.xxx.xx:9292            |          http://xxx.xxx.xxx.xx:9292         | df971f50591c40ff974aa535a2c9138c |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+

More information. First, the keystone log from the initial login of usr1:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running the openstack services with the exception of .2 which is running my dashboard. I guess those failed authorizations from .41 are Nova? Maybe getting close.

Next, the following settings are in my nova.conf for the keystone_authtoken section. I guess the commented-out values are the defaults, and the remaining ones are overrides:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive
click to hide/show revision 5
No.5 Revision

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up multi-domain support, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:

DEBUG = True
OPENSTACK_API_VERSIONS = {
    "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea is that the UI should know what different users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.


In response to mpetason, here is the service list:

# keystone service-list
+----------------------------------+------------+--------------+--------------------------------+
|                id                |    name    |     type     |          description           |
+----------------------------------+------------+--------------+--------------------------------+
| b289d7f1545a49d2a35e64db07ab9c3b | ceilometer |   metering   |   Openstack Metering Service   |
| 2273ffaebf3949bc8460365e78e190d0 |   cinder   |    volume    |         Cinder Service         |
| 08cb46b56b8546beb62bc8f2e1426ed9 | cinder_v2  |   volumev2   |       Cinder Service v2        |
| e5d03c64d2b14b9db55c0ce60f9debca |  cinderv2  |   volumev2   |       Cinder Service v2        |
| df971f50591c40ff974aa535a2c9138c |   glance   |    image     |    Openstack Image Service     |
| c95586316d1f41ab9af05dd7efd09bbb |  keystone  |   identity   |   OpenStack Identity Service   |
| 57ec26fda6ae4596b88a42c0a32a5d78 |  neutron   |   network    |   Neutron Networking Service   |
| 23ff0372d10c4396831a22c5e0a25f93 |    nova    |   compute    |   Openstack Compute Service    |
| cd32c1f660b349cdaa16ef1ff9a2bd54 |  nova_ec2  |     ec2      |          EC2 Service           |
| c7c107af6d014e69955f9b0b9d7ec0e6 |   swift    | object-store | Openstack Object-Store Service |
| 1b7f416c5a2d48e5a96728e210bcb770 |  swift_s3  |      s3      |      Openstack S3 Service      |
+----------------------------------+------------+--------------+--------------------------------+

and the endpoint-list

# keystone endpoint-list
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
|                id                |   region  |                    publicurl                     |                   internalurl                    |                   adminurl                  |            service_id            |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
| 48ac9fde57b5433a84f1dfff7c96fe04 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | e5d03c64d2b14b9db55c0ce60f9debca |
| 507a6b832402452b90aa7fb233683259 | RegionOne |         http://xxx.xxx.xxx.xx:5000/v2.0          |         http://xxx.xxx.xxx.xx:5000/v2.0          |       http://xxx.xxx.xxx.xx:35357/v2.0      | c95586316d1f41ab9af05dd7efd09bbb |
| 789c19eb25174c94945337621c2143df | RegionOne |            http://xxx.xxx.xxx.xx:8080            |            http://xxx.xxx.xxx.xx:8080            |          http://xxx.xxx.xxx.xx:8080         | 1b7f416c5a2d48e5a96728e210bcb770 |
| 7c39239bdbd94af48836a09b093c1b1e | RegionOne |            http://xxx.xxx.xxx.xx:8777            |            http://xxx.xxx.xxx.xx:8777            |          http://xxx.xxx.xxx.xx:8777         | b289d7f1545a49d2a35e64db07ab9c3b |
| 985c9cf36c784c16a54bdb8c9520c687 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | 08cb46b56b8546beb62bc8f2e1426ed9 |
| a325af2a574b41308ae7cf4e5d234f13 | RegionOne |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |  http://xxx.xxx.xxx.xx:8773/services/Admin  | cd32c1f660b349cdaa16ef1ff9a2bd54 |
| a39532546fcd4a26ab0c027c0a920269 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s | 2273ffaebf3949bc8460365e78e190d0 |
| be86120cd43245fea4c412d0c8ad6c34 | RegionOne | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s |         http://xxx.xxx.xxx.xx:8080/         | c7c107af6d014e69955f9b0b9d7ec0e6 |
| bfd29a18db84492e81c3adc5ab6bcbf3 | RegionOne |           http://xxx.xxx.xxx.xx:9696/            |           http://xxx.xxx.xxx.xx:9696/            |         http://xxx.xxx.xxx.xx:9696/         | 57ec26fda6ae4596b88a42c0a32a5d78 |
| cf75ad45a0cf4d42aea2e56c8d52c588 | RegionOne |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s | 23ff0372d10c4396831a22c5e0a25f93 |
| df8dbbe9feb546498b19cb2976daa724 | RegionOne |            http://xxx.xxx.xxx.xx:9292            |            http://xxx.xxx.xxx.xx:9292            |          http://xxx.xxx.xxx.xx:9292         | df971f50591c40ff974aa535a2c9138c |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+

More information. First, the keystone log from the initial login of usr1:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running the openstack services with the exception of .2 which is running my dashboard. I guess those failed authorizations from .41 are Nova? Maybe getting close.

Next, the following settings are in my nova.conf for the keystone_authtoken section. I guess the commented-out values are the defaults, and the remaining ones are overrides:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

IceHouse: dashboard problems using multi-domain support

I originally asked this question on stackoverflow, and then realized this forum is probably more appropriate...

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc.

To set up etc. When multi-domain support, dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients.

In the original post linked above, I was playing with an openstack-dashboard using clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this also doesn't work for me, it is off-topic.

Since then I have reverted to a bleeding-edge version from the GitHub repo. However, I thought it would be a good idea to use the same vintage for all of the services. So I am now using the RDO/IceHouse install of the dashboard which is slightly older instead.

First, I modified the following lines in /etc/openstack-dashboard/local_settings:more basic setup to simplify things. Leaving everything else in the default state, I enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

DEBUG = True
OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

I then created soft links from /etc/openstack-dashboard/[nova_|keystone_|glance_|cinder_].json to /etc/[service]/policy.json. With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains etc.

The main thing is getting the updated keystone/policy.json file with v3 support enabled (note that I took the template v3 policy file from the IceHouse branch in the GitHub repo, being sure to replace the string 'admin_domain_id' with the actual admin_domain ID). I'm not sure how important this step is - I guess the idea problem is that the UI should know what different when one of these domain users are able to see, rather than just blindly feeding through REST calls and getting auth errors.

I restart my web server, and now the dashboard requests a domain in addition to username/password. logs in, they cannot access any of the services. The initial indication of this problem is when nova is queried in order to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, I suppose it is trying to collect information from nova for the overview page:problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

Unlike my original test (using dashboard from the repo and the dev environment + lightweight Django web server), which still allowed me to view the pages, but with red bubbles saying "Error: Unauthorized..." I just get an "Unauthorized at /project/" web page with the call stack shown above. I'm guessing this is just down to differences in the dashboard and/or apache configuration. At any rate, my best guess so far is that I have a v3 domain-scoped token, but it's trying to use the v2 API with Nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"

My goal here isn't to get a fully-functional dashboard for admin users, but I would like the normal user experience to be reasonably good, and my impression from reading scant reports around the web that this should be possible with IceHouse.

Any help would be greatly appreciated.


In response to mpetason, here is the service list:

# keystone service-list
+----------------------------------+------------+--------------+--------------------------------+
|                id                |    name    |     type     |          description           |
+----------------------------------+------------+--------------+--------------------------------+
| b289d7f1545a49d2a35e64db07ab9c3b | ceilometer |   metering   |   Openstack Metering Service   |
| 2273ffaebf3949bc8460365e78e190d0 |   cinder   |    volume    |         Cinder Service         |
| 08cb46b56b8546beb62bc8f2e1426ed9 | cinder_v2  |   volumev2   |       Cinder Service v2        |
| e5d03c64d2b14b9db55c0ce60f9debca |  cinderv2  |   volumev2   |       Cinder Service v2        |
| df971f50591c40ff974aa535a2c9138c |   glance   |    image     |    Openstack Image Service     |
| c95586316d1f41ab9af05dd7efd09bbb |  keystone  |   identity   |   OpenStack Identity Service   |
| 57ec26fda6ae4596b88a42c0a32a5d78 |  neutron   |   network    |   Neutron Networking Service   |
| 23ff0372d10c4396831a22c5e0a25f93 |    nova    |   compute    |   Openstack Compute Service    |
| cd32c1f660b349cdaa16ef1ff9a2bd54 |  nova_ec2  |     ec2      |          EC2 Service           |
| c7c107af6d014e69955f9b0b9d7ec0e6 |   swift    | object-store | Openstack Object-Store Service |
| 1b7f416c5a2d48e5a96728e210bcb770 |  swift_s3  |      s3      |      Openstack S3 Service      |
+----------------------------------+------------+--------------+--------------------------------+

and the endpoint-list

# keystone endpoint-list
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
|                id                |   region  |                    publicurl                     |                   internalurl                    |                   adminurl                  |            service_id            |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+
| 48ac9fde57b5433a84f1dfff7c96fe04 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | e5d03c64d2b14b9db55c0ce60f9debca |
| 507a6b832402452b90aa7fb233683259 | RegionOne |         http://xxx.xxx.xxx.xx:5000/v2.0          |         http://xxx.xxx.xxx.xx:5000/v2.0          |       http://xxx.xxx.xxx.xx:35357/v2.0      | c95586316d1f41ab9af05dd7efd09bbb |
| 789c19eb25174c94945337621c2143df | RegionOne |            http://xxx.xxx.xxx.xx:8080            |            http://xxx.xxx.xxx.xx:8080            |          http://xxx.xxx.xxx.xx:8080         | 1b7f416c5a2d48e5a96728e210bcb770 |
| 7c39239bdbd94af48836a09b093c1b1e | RegionOne |            http://xxx.xxx.xxx.xx:8777            |            http://xxx.xxx.xxx.xx:8777            |          http://xxx.xxx.xxx.xx:8777         | b289d7f1545a49d2a35e64db07ab9c3b |
| 985c9cf36c784c16a54bdb8c9520c687 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v2/%(tenant_id)s | 08cb46b56b8546beb62bc8f2e1426ed9 |
| a325af2a574b41308ae7cf4e5d234f13 | RegionOne |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |    http://xxx.xxx.xxx.xx:8773/services/Cloud     |  http://xxx.xxx.xxx.xx:8773/services/Admin  | cd32c1f660b349cdaa16ef1ff9a2bd54 |
| a39532546fcd4a26ab0c027c0a920269 | RegionOne |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8776/v1/%(tenant_id)s | 2273ffaebf3949bc8460365e78e190d0 |
| be86120cd43245fea4c412d0c8ad6c34 | RegionOne | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s | http://xxx.xxx.xxx.xx:8080/v1/AUTH_%(tenant_id)s |         http://xxx.xxx.xxx.xx:8080/         | c7c107af6d014e69955f9b0b9d7ec0e6 |
| bfd29a18db84492e81c3adc5ab6bcbf3 | RegionOne |           http://xxx.xxx.xxx.xx:9696/            |           http://xxx.xxx.xxx.xx:9696/            |         http://xxx.xxx.xxx.xx:9696/         | 57ec26fda6ae4596b88a42c0a32a5d78 |
| cf75ad45a0cf4d42aea2e56c8d52c588 | RegionOne |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    |   http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s    | http://xxx.xxx.xxx.xx:8774/v2/%(tenant_id)s | 23ff0372d10c4396831a22c5e0a25f93 |
| df8dbbe9feb546498b19cb2976daa724 | RegionOne |            http://xxx.xxx.xxx.xx:9292            |            http://xxx.xxx.xxx.xx:9292            |          http://xxx.xxx.xxx.xx:9292         | df971f50591c40ff974aa535a2c9138c |
+----------------------------------+-----------+--------------------------------------------------+--------------------------------------------------+---------------------------------------------+----------------------------------+

More information. First, the The keystone log from the initial login of usr1:shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running the openstack services with the exception of .2 which is running my dashboard. I guess those failed authorizations from .41 are Nova? Maybe getting close.

Next, the following settings are in my nova.conf for Nova?

To be clear, I don't get these problems when entering as the admin user.

There was a suggestion from mpetason to add the v3 endpoints explicitly to keystone. This did not seem to make any difference.

Another suggestion from him was to check the keystone_authtoken section. I guess the commented-out values are the defaults, and the remaining ones are overrides:section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state, I enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains etc.

The problem is that when one of these domain users logs in, they cannot access any of the services. do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried in order to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services services, with the exception of .2 the dashboard which is running my dashboard. on the machine that ends with .2. I guess those failed authorizations from .41 are Nova?

To be clear, I don't get these problems when entering as the admin user.

There was a suggestion from mpetason to add the v3 endpoints explicitly to keystone. This did not seem to make any difference.

Another suggestion from him was to check the keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state, I state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova?

To be clear, I don't get these problems when entering as the admin user.

There was a suggestion from mpetason to add the v3 endpoints explicitly to keystone. This did not seem to make any difference.

Another suggestion from him was to check the keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova?Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

There was a suggestion from Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason to add the v3 endpoints explicitly to keystone. This did not seem to make any difference.

Another suggestion from him was to check the suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users work. Interestingly, a normal user in the default domain works fine.

    • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users work. Interestingly, a normal user in the default domain works fine.

    • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those demains domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain useruser problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).

Any further suggestions would be greatly appreciated.

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).


Any further suggestions would be greatly appreciated.Suggestions in update 2 from Haneef:

  • I have changed keystone.conf to set provider = keystone.token.providers.uuid.Provider
  • I request a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.
  • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"
  • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

this is the keystone log:

2014-08-29 14:01:55.873 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.874 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.006418
2014-08-29 14:01:55.966 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "POST /v2.0/tokens HTTP/1.1" 200 4073 0.090002
2014-08-29 14:01:55.973 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.974 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.005451

and here is the nova log:

2014-08-29 14:01:55.867 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-29 14:01:55.876 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.968 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.974 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.974 10634 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-29 14:01:55.975 10634 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-29 14:01:55.975 10634 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-29 14:01:55.975 10634 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.41 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1091890
  • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

  • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).


Suggestions in update 2 from Haneef:

  • I have changed keystone.conf to set provider = keystone.token.providers.uuid.Provider

  • I request a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.

  • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"

  • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

this is the keystone log:

2014-08-29 14:01:55.873 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.874 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.006418
2014-08-29 14:01:55.966 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "POST /v2.0/tokens HTTP/1.1" 200 4073 0.090002
2014-08-29 14:01:55.973 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.974 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.005451

and here is the nova log:

2014-08-29 14:01:55.867 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-29 14:01:55.876 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.968 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.974 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.974 10634 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-29 14:01:55.975 10634 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-29 14:01:55.975 10634 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-29 14:01:55.975 10634 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.41 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1091890
  • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

  • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).


Suggestions in update 2 from Haneef:

  • I have changed keystone.conf to set provider = keystone.token.providers.uuid.Provider

  • I request a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.

  • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"

  • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

this is the keystone log:

2014-08-29 14:01:55.873 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.874 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.006418
2014-08-29 14:01:55.966 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "POST /v2.0/tokens HTTP/1.1" 200 4073 0.090002
2014-08-29 14:01:55.973 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.974 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.005451

and here is the nova log:

2014-08-29 14:01:55.867 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-29 14:01:55.876 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.968 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.974 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.974 10634 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-29 14:01:55.975 10634 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-29 14:01:55.975 10634 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-29 14:01:55.975 10634 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.41 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1091890
  • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

  • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json


Suggestions in update 3 from Haneef:

  • setting debug=True I get the detail that he expected:

    2014-08-29 14:54:39.805 4006 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported from 132.246.194.41

    • I now edit nova.conf and set auth_version=v3.0.

My simple curl test works!

I had actually tried setting auth_version=v3 previously and this had not worked

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).


Suggestions in update 2 from Haneef:

  • I have changed keystone.conf to set provider = keystone.token.providers.uuid.Provider

  • I request a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.

  • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"

  • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

this is the keystone log:

2014-08-29 14:01:55.873 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.874 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.006418
2014-08-29 14:01:55.966 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "POST /v2.0/tokens HTTP/1.1" 200 4073 0.090002
2014-08-29 14:01:55.973 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.974 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.005451

and here is the nova log:

2014-08-29 14:01:55.867 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-29 14:01:55.876 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.968 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.974 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.974 10634 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-29 14:01:55.975 10634 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-29 14:01:55.975 10634 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-29 14:01:55.975 10634 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.41 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1091890
  • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

  • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json


Suggestions in update 3 from Haneef:

  • setting debug=True I get the detail that he expected:

    2014-08-29 14:54:39.805 4006 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported from 132.246.194.41

    • I now edit nova.conf and set auth_version=v3.0.

My simple curl / nova test works!

I had actually tried setting auth_version=v3 previously and this had not worked


Now that nova seems to be working, on to other services.

There are auth_version parameters which I have set as above in ceilometer.conf and cinder.conf.

What about other services?

For example, glance has multiple config files, none of which contain auth_version. I've tried setting auth_url=http://132.246.194.41:5000/v3, but I can still clearly see v2 auth in the logs (and it doesn't work). Thinking auth_version may be an undocumented parameter in the [keystone_authtoken] section of glance-api.conf (see this bug report I try setting it, and get different errors.

keystone log:

2014-08-29 16:39:57.810 1310 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/site-packages/keystone/common/controller.py:151

2014-08-29 16:39:57.814 1310 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 16:39:57] "GET /v3/auth/tokens HTTP/1.1" 200 6724 0.013372

glance api log:

2014-08-29 16:39:57.799 1666 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 16:39:57.814 1666 DEBUG urllib3.connectionpool [-] "GET /v3/auth/tokens HTTP/1.1" 200 6543 _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:357
2014-08-29 16:39:57.816 1666 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache _cache_put /usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1234
2014-08-29 16:39:57.816 1666 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: f4f18d9d90504b90b47093976de4aa49 with project_id : 0c2af5ec1f904640abd0c49e1547e57d and roles: _member_  _build_user_headers /usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1021
2014-08-29 16:39:57.823 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Matched GET /images/detail __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:100
2014-08-29 16:39:57.824 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Route path: '/images/detail', defaults: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x241e550>} __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:102
2014-08-29 16:39:57.824 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Match dict: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x241e550>} __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:103
2014-08-29 16:39:57.825 1666 DEBUG glance.common.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=created_at _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:411
2014-08-29 16:39:57.829 1666 DEBUG glance.common.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=created_at _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:411
2014-08-29 16:39:57.832 1666 INFO glance.registry.client.v1.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Registry client request GET /images/detail raised NotAuthenticated
2014-08-29 16:39:57.835 1666 INFO glance.wsgi.server [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Traceback (most recent call last):
[...shortened...]
  File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 386, in do_request
    headers=copy.deepcopy(headers))
  File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 83, in wrapped
    return func(self, method, url, body, headers)
  File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 527, in _do_request
    raise exception.NotAuthenticated(res.read())
NotAuthenticated: Authentication required
2014-08-29 16:39:57.835 1666 INFO glance.wsgi.server [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] 132.246.194.41 - - [29/Aug/2014 16:39:57] "GET /v1/images/detail?sort_key=created_at HTTP/1.1" 500 139 0.259652

IceHouse: dashboard problems using multi-domain support

I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

OPENSTACK_API_VERSIONS = {
     "identity": 3
}
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"

With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".

Next, problems with nova:

2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
    handled = self.construct_tables()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
    handled = self.handle_table(table)
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
    data = self._get_data_dict()
  File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
    super(ProjectOverview, self).get_data()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
    self.usage.summarize(*self.usage.get_date_range())
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
    if not api.nova.extension_supported('SimpleTenantUsage', self.request):
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
    extensions = list_extensions(request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
    return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
  File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
    return self._list("/extensions", 'extensions')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)

The keystone log shows things like this:

2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461

The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

 2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861

To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

[keystone_authtoken]
#auth_admin_prefix=
auth_host=xxx.xxx.xxx.41
#auth_port=35357
auth_protocol=http
auth_uri=http://xxx.xxx.xxx.41:5000/
#auth_version=v2.0
#delay_auth_decision=false
#http_connect_timeout=<None>
#http_request_max_retries=3
#http_handler=<None>
#admin_token=<None>
admin_user=nova
admin_password=xxxxxx
admin_tenant_name=services
#cache=<None>
#certfile=<None>
#keyfile=<None>
#cafile=<None>
#insecure=false
#signing_dir=<None>
#memcached_servers=<None>
#token_cache_time=300
#revocation_cache_time=1
#memcache_security_strategy=<None>
#memcache_secret_key=<None>
#include_service_catalog=true
#enforce_token_bind=permissive

Other Some things I have looked at:

  • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

  • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.


The correct solution came from Haneef

  • Also, in response to Haneef, I have verified that I can obtain a v3 tokens for a new domain user, e.g.: TOKEN=$( curl http://x.x.x.x:5000/v3/auth/tokens -s -i -H "Content-Type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": { "user": { "domain": { "name": "foo" }, "name": "echapin", "password": "password" } } } } }' | grep ^X-Subject-Token: | awk '{print $2}' ) works. This user has the _member_ roll on a project in that domain (I set this up in the dashboard when using the normal policy.json).


Suggestions in update 2 from Haneef:

  • I have I changed keystone.conf to set provider = keystone.token.providers.uuid.Provider

  • I request requested a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.

  • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"

  • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

this is the keystone log:

2014-08-29 14:01:55.873 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.874 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.006418
2014-08-29 14:01:55.966 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "POST /v2.0/tokens HTTP/1.1" 200 4073 0.090002
2014-08-29 14:01:55.973 10556 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 132.246.194.41
2014-08-29 14:01:55.974 10556 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 14:01:55] "GET /v2.0/tokens/a2075b9fd43d40fea0c200847ed6290d HTTP/1.1" 401 315 0.005451

and here is the nova log:

2014-08-29 14:01:55.867 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.875 10634 INFO keystoneclient.middleware.auth_token [-] Retrying validation
2014-08-29 14:01:55.876 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.968 10634 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
2014-08-29 14:01:55.974 10634 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
2014-08-29 14:01:55.974 10634 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2014-08-29 14:01:55.975 10634 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2014-08-29 14:01:55.975 10634 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
2014-08-29 14:01:55.975 10634 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.41 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1091890
  • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

  • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json


Suggestions in update 3

  • setting debug=True in keystone.conf, I get this valuable detail from Haneef:

    • setting debug=True I get the detail that he expected:the keystone log which is the smoking gun of the wrong auth API being used internally by nova:

      2014-08-29 14:54:39.805 4006 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported from 132.246.194.41

      • I now edit nova.conf and set auth_version=v3.0. and the simple nova / curl test works. Note that I tried something like this before, but I was caught by the v3.0 instead of v3 as it appears everywhere else

    My simple curl / nova test works!

    I had actually tried setting auth_version=v3 previously and this had not worked


    Now that nova seems to be working, on

  • The dashboard still has numerous problems due to other services.

    There are auth_version parameters which I have set as above in ceilometer.conf services requiring this same configuration change. Ultimately I had to find the [keystone_authtoken] sections in all of the following files, and cinder.conf.

    What about other services?

    For example, glance has multiple config files, none of which contain auth_version. I've tried setting auth_url=http://132.246.194.41:5000/v3, but I can still clearly see v2 auth in the logs (and add auth_version=v3.0 (in some cases it doesn't work). Thinking auth_version may appears to be an undocumented parameter in the [keystone_authtoken] section value): nova/nova.conf, ceilometer/ceilometer.conf, cinder/cinder.conf, glance/glance-api.conf, glance/glance-registry.conf, and neutron/neutron.conf.

  • After doing this (and restarting the OpenStack services), a user from a newly-created domain can use most of glance-api.conf (see this bug report I try setting it, and get different errors.

    keystone log:

    2014-08-29 16:39:57.810 1310 DEBUG keystone.common.controller [-] RBAC: Authorization granted inner /usr/lib/python2.7/site-packages/keystone/common/controller.py:151
    

    2014-08-29 16:39:57.814 1310 INFO eventlet.wsgi.server [-] 132.246.194.41 - - [29/Aug/2014 16:39:57] "GET /v3/auth/tokens HTTP/1.1" 200 6724 0.013372

    glance api log:

    2014-08-29 16:39:57.799 1666 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-29 16:39:57.814 1666 DEBUG urllib3.connectionpool [-] "GET /v3/auth/tokens HTTP/1.1" 200 6543 _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:357
    2014-08-29 16:39:57.816 1666 DEBUG keystoneclient.middleware.auth_token [-] Storing token in cache _cache_put /usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1234
    2014-08-29 16:39:57.816 1666 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: f4f18d9d90504b90b47093976de4aa49 with project_id : 0c2af5ec1f904640abd0c49e1547e57d and roles: _member_  _build_user_headers /usr/lib/python2.7/site-packages/keystoneclient/middleware/auth_token.py:1021
    2014-08-29 16:39:57.823 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Matched GET /images/detail __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:100
    2014-08-29 16:39:57.824 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Route path: '/images/detail', defaults: {'action': u'detail', 'controller': <glance.common.wsgi.Resource the dashboard functionality. The only exception I haven't gotten to work yet is the object at 0x241e550>} __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:102
    2014-08-29 16:39:57.824 1666 DEBUG routes.middleware [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Match dict: {'action': u'detail', 'controller': <glance.common.wsgi.Resource object at 0x241e550>} __call__ /usr/lib/python2.7/site-packages/routes/middleware.py:103
    2014-08-29 16:39:57.825 1666 DEBUG glance.common.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=created_at _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:411
    2014-08-29 16:39:57.829 1666 DEBUG glance.common.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Constructed URL: http://0.0.0.0:9191/images/detail?sort_key=created_at _construct_url /usr/lib/python2.7/site-packages/glance/common/client.py:411
    2014-08-29 16:39:57.832 1666 INFO glance.registry.client.v1.client [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Registry client request GET /images/detail raised NotAuthenticated
    2014-08-29 16:39:57.835 1666 INFO glance.wsgi.server [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] Traceback (most recent call last):
    [...shortened...]
      File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 386, in do_request
        headers=copy.deepcopy(headers))
      File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 83, in wrapped
        return func(self, method, url, body, headers)
      File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 527, in _do_request
        raise exception.NotAuthenticated(res.read())
    NotAuthenticated: Authentication required
    2014-08-29 16:39:57.835 1666 INFO glance.wsgi.server [563540d1-4ea9-4ccf-8af7-f868047d0095 f4f18d9d90504b90b47093976de4aa49 0c2af5ec1f904640abd0c49e1547e57d - - -] 132.246.194.41 - - [29/Aug/2014 16:39:57] "GET /v1/images/detail?sort_key=created_at HTTP/1.1" 500 139 0.259652
    
    store.

    IceHouse: dashboard problems using multi-domain support

    I am prototyping multi-domain support using an "out-of-the-box" RDO installation of IceHouse on a single machine. Using the default single-domain settings, I have tested and verified that admin and regular users can both log in to the dashboard, view images, launch instances etc. When multi-domain dashboard support is activated, users cannot access any of the services (nova, glance etc.)

    In the intial version of this question, I followed this guide to create the cloud_admin user, two domains (dom1, and dom2), as well as a domain administrator (adm1) and user (usr1) on dom1. This is all done using curl rather than python clients. I now realize that the purpose of this work isn't necessarily to enable basic multi-domain support, rather the ability to delegate administration of domains to special domain administrators. While this feature also doesn't work for me, it is off-topic.

    Since then I have reverted to a more basic setup to simplify things. Leaving everything else in the default state (including the single-domain keystone/policy.json), I only enable multi-domain support in the dashboard, e.g., in /etc/openstack-dashboard:

    OPENSTACK_API_VERSIONS = {
         "identity": 3
    }
    OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
    OPENSTACK_KEYSTONE_URL = "http://x.x.x.x:5000/v3"
    

    With this, I can now log in as the admin user on the "default" domain, create new domains, users and projects in those domains etc.

    The problem is that when one of these domain users logs in, they cannot do anything (e.g., list images, launch a VM). The initial indication of this problem is when nova is queried to generate the summary page.

    I enter "dom1", "usr1" and "password_for_usr1". In the dashboard log I can see that the user correctly authenticated:

    2014-08-25 18:27:14,039 7259 DEBUG openstack_auth.backend Authentication completed for user "usr1".
    2014-08-25 18:27:14,039 7259 INFO openstack_auth.forms Login successful for user "usr1".
    

    Next, problems with nova:

    2014-08-25 18:27:14,093 7260 DEBUG openstack_dashboard.api.nova novaclient connection created using token "xxxxx" and url "http://132.246.194.41:8774/v2/xxxxx"
    2014-08-25 18:27:14,330 7260 ERROR django.request Internal Server Error: /dashboard/project/
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
        response = wrapped_callback(request, *callback_args, **callback_kwargs)
      File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
        return view_func(request, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 54, in dec
        return view_func(request, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 38, in dec
        return view_func(request, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
        return self.dispatch(request, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
        return handler(request, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 154, in get
        handled = self.construct_tables()
      File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 145, in construct_tables
        handled = self.handle_table(table)
      File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 118, in handle_table
        data = self._get_data_dict()
      File "/usr/lib/python2.7/site-packages/horizon/tables/views.py", line 181, in _get_data_dict
        self._data = {self.table_class._meta.name: self.get_data()}
      File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/overview/views.py", line 57, in get_data
        super(ProjectOverview, self).get_data()
      File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/views.py", line 43, in get_data
        self.usage.summarize(*self.usage.get_date_range())
      File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/base.py", line 200, in summarize
        if not api.nova.extension_supported('SimpleTenantUsage', self.request):
      File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
        value = cache[key] = func(*args, **kwargs)
      File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 752, in extension_supported
        extensions = list_extensions(request)
      File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
        value = cache[key] = func(*args, **kwargs)
      File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/nova.py", line 743, in list_extensions
        return nova_list_extensions.ListExtManager(novaclient(request)).show_all()
      File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/list_extensions.py", line 37, in show_all
        return self._list("/extensions", 'extensions')
      File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 64, in _list
        _resp, body = self.api.client.get(url)
      File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 309, in get
        return self._cs_request(url, 'GET', **kwargs)
      File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request
        raise e
    Unauthorized: Unauthorized (HTTP 401)
    

    The keystone log shows things like this:

    2014-08-27 16:27:15.873 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:15] "POST /v3/auth/tokens HTTP/1.1" 201 16000 0.125059
    2014-08-27 16:27:16.051 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.110675
    2014-08-27 16:27:16.059 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
    2014-08-27 16:27:16.059 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006470
    2014-08-27 16:27:16.159 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "POST /v2.0/tokens HTTP/1.1" 200 9869 0.097592
    2014-08-27 16:27:16.166 2447 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from xxx.xxx.xxx.41
    2014-08-27 16:27:16.167 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.41 - - [27/Aug/2014 16:27:16] "GET /v2.0/tokens/1b15bf2b1ff6f4f60ee4e032316030e7 HTTP/1.1" 401 315 0.006236
    2014-08-27 16:27:16.276 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "POST /v3/auth/tokens HTTP/1.1" 201 16021 0.070850
    2014-08-27 16:27:16.291 2447 INFO eventlet.wsgi.server [-] xxx.xxx.xxx.2 - - [27/Aug/2014 16:27:16] "GET /v3/users/ad9f2582da994e5793a4c5100d9ca78a/projects HTTP/1.1" 200 514 0.009461
    

    The IP address ending in .41 is the machine running most of the openstack services, with the exception of the dashboard which is running on the machine that ends with .2. I guess those failed authorizations from .41 are Nova. If we check the nova log (from a separate run, time-stamps don't match) we see things like this:

     2014-08-28 15:20:57.748 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
    2014-08-28 15:20:57.756 21461 INFO keystoneclient.middleware.auth_token [-] Retrying validation
    2014-08-28 15:20:57.757 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:57.883 21461 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
    2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
    2014-08-28 15:20:57.891 21461 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
    2014-08-28 15:20:57.891 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
    2014-08-28 15:20:57.891 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.1441140
    2014-08-28 15:20:57.895 21461 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
    2014-08-28 15:20:57.895 21461 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
    2014-08-28 15:20:57.895 21461 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004969
    2014-08-28 15:20:57.900 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:58.000 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
    2014-08-28 15:20:58.008 21463 INFO keystoneclient.middleware.auth_token [-] Retrying validation
    2014-08-28 15:20:58.009 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:58.108 21463 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 132.246.194.41
    2014-08-28 15:20:58.116 21463 INFO keystoneclient.middleware.auth_token [-] Keystone rejected admin token, resetting
    2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Invalid user token. Keystone response: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
    2014-08-28 15:20:58.116 21463 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
    2014-08-28 15:20:58.117 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
    2014-08-28 15:20:58.117 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d/extensions HTTP/1.1" status: 401 len: 197 time: 0.2179890
    2014-08-28 15:20:58.122 21463 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
    2014-08-28 15:20:58.122 21463 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
    2014-08-28 15:20:58.122 21463 INFO nova.osapi_compute.wsgi.server [-] 132.246.194.2 "GET /v2/0c2af5ec1f904640abd0c49e1547e57d HTTP/1.1" status: 401 len: 197 time: 0.0004861
    

    To be clear, I don't get these problems when entering as the admin user. Things also work fine with a normal user in the default domain.

    Note the errors in the nova log about "keystoneclient.middleware.auth_token". mpetason suggested checking keystone_authtoken section in nova.conf. I'm not sure what I'm looking for, but this is what I have:

    [keystone_authtoken]
    #auth_admin_prefix=
    auth_host=xxx.xxx.xxx.41
    #auth_port=35357
    auth_protocol=http
    auth_uri=http://xxx.xxx.xxx.41:5000/
    #auth_version=v2.0
    #delay_auth_decision=false
    #http_connect_timeout=<None>
    #http_request_max_retries=3
    #http_handler=<None>
    #admin_token=<None>
    admin_user=nova
    admin_password=xxxxxx
    admin_tenant_name=services
    #cache=<None>
    #certfile=<None>
    #keyfile=<None>
    #cafile=<None>
    #insecure=false
    #signing_dir=<None>
    #memcached_servers=<None>
    #token_cache_time=300
    #revocation_cache_time=1
    #memcache_security_strategy=<None>
    #memcache_secret_key=<None>
    #include_service_catalog=true
    #enforce_token_bind=permissive
    

    Some things I looked at:

    • mpetason suggested adding the v3 endpoints explicitly to keystone. I did this and saw no difference

    • I tried switching to policy.v3cloudsample.json for keystone (from the head of the IceHouse branch in the GitHub repo), and setting "admin_domain_id = default" as suggested by Haneef. In addition to not solving the new domain user problem, the admin user now has auth problems in the dashboard. None of list_domains, list_groups, list_users etc. work. Interestingly, a normal user in the default domain works fine.


    The correct solution came from Haneef

    • I changed keystone.conf to set provider = keystone.token.providers.uuid.Provider this is purely for testing purposes, and was removed once things worked

    • I requested a project-scoped token for a user in one of my new domains (e.g., using a "scope": { "project": {... in my curl command to the identity service). I can clearly see the role _member_ listed in the response.

    • A simple test of this project-scoped token with keystone works, e.g., curl -i -X GET http://132.246.194.41:5000/v3/users/$ID_USER/projects -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $USER_TOKEN"

    • The nova API call that Horizon is doing fails: curl -i "http://132.246.194.41:8774/v2/$ID_PROJECT/extensions" -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: $ID_PROJECT" -H "X-Auth-Token: $USER_TOKEN" with a response HTTP/1.1 401 Unauthorized.

    • if I repeat this test using a project scoped token for a regular user (_member_ role) in the default domain everything works

    • I see the same behaviour using both the default policy.json for keystone and policy.v3cloudsample.json

    • setting debug=True in keystone.conf, I get this valuable detail from the keystone log which is the smoking gun of the wrong auth API being used internally by nova:

      2014-08-29 14:54:39.805 4006 WARNING keystone.common.wsgi [-] Authorization failed. Non-default domain is not supported from 132.246.194.41

    • I now edit nova.conf and set auth_version=v3.0 and the simple nova / curl test works. Note that I tried something like this before, but I was caught by the v3.0 instead of v3 as it appears everywhere else

    • The dashboard still has numerous problems due to other services requiring this same configuration change. Ultimately I had to find the [keystone_authtoken] sections in all of the following files, and add auth_version=v3.0 (in some cases it appears to be an undocumented value): nova/nova.conf, ceilometer/ceilometer.conf, cinder/cinder.conf, glance/glance-api.conf, glance/glance-registry.conf, and neutron/neutron.conf.

    After doing this (and restarting the OpenStack services), a user from a newly-created domain can use most of the dashboard functionality. The only exception I haven't gotten to work yet is the object store.