Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

neutron client authentication failed

I am running Havana/Fedora 19 and my nova-compute reports neutron authentication failure. It indicates that the service or the endpoint are not available. I posted a request on the launchpad URL and I got a suggestion to include neutron_region_name in the nova.conf file. I included it on both the controller and the compute nodes and restarted all my services, but I am still running into the same error.

Since I am using RDO packages, I thought I'd post it it on the RDO forum to see if anyone can point me in the right direction. A copy-paste of my question from the launchpad site is attached below.

=========BEGIN COPY/PASTE===========

I am running Havana on a Fedora 19 system and I am running into some issues in nova-compute when booting an instance. I am getting a "neutron client authentication failed: Could not find Service or Region in Service Catalog" error in the nova-compute log file (note 1). However, the endpoints do exist when run from the controller node (note 2). I am also able to use the neutron client to print previously created networks and all other related information (note 3), so I am not quite sure if it's the endpoint that is causing issues. Nova.conf appears to have the right settings to indicate where the neutron is running (note 4) while the api-paste.ini file also the right information (note 5). To verify, I ran keystone commands from the compute host and I am able to see the endpoints (note 6).

It does not appear to be the issue described in https://bugs.launchpad.net/nova/+bug/1241275 since the stack traces are different. From the looks of it, in my case, there is an attempt to get the token and it fails to locate even the endpoint. Any thoughts or suggestions in this regard are greatly appreciated.

Notes:

  1. nova-compute log

2013-11-26 14:02:56.921 2493 ERROR nova.network.neutronv2 [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Neutron client authentication failed: Could not find Service or Region in Service Catalog. 2013-11-26 14:02:56.922 2493 ERROR nova.compute.manager [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Instance failed network setup after 1 attempt(s) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager Traceback (most recent call last): 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager dhcp_options=dhcp_options) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapper 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager res = f(self, context, args, *kwargs) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.exception(msg, port_id) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 323, in allocate_for_instance 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager port_req_body) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 392, in _populate_neutron_extension_values 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._refresh_neutron_extensions_cache() 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 375, in _refresh_neutron_extensions_cache 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager neutron = neutronv2.get_client(context.get_admin_context()) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 71, in get_client 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager return _get_client(token=token) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 52, in _get_client 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager token = _get_auth_token() 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 47, in _get_auth_token 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.error(_('Neutron client authentication failed: %s'), e) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 43, in _get_auth_token 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager httpclient.authenticate() 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 245, in authenticate 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._extract_service_catalog(body) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 208, in _extract_service_catalog 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager endpoint_type=self.endpoint_type) 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 82, in url_for 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager raise exceptions.EndpointNotFound() 2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager EndpointNotFound: Could not find Service or Region in Service Catalog.

Note 2:

[root@drogon utils]# keystone endpoint-list +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 | | 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf | | 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 | | d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 | | d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 | | f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac | | fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

Note 3: Printing net-list as both admin and demo users

[root@viserion utils]# neutron net-list +--------------------------------------+----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+--------------------------------------------------+ | 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 | | eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 7.7.7.0/24 | +--------------------------------------+----------+--------------------------------------------------+

[root@viserion utils]# . ./keystone_demo_remote.bash [root@viserion utils]# neutron net-list +--------------------------------------+----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+--------------------------------------------------+ | 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 | | eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 | +--------------------------------------+----------+--------------------------------------------------+

Note 4: excerpt from nova.conf file on compute host

network_api_class = nova.network.neutronv2.api.API neutron_url = http://192.168.122.61:9696 neutron_auth_strategy = keystone neutron_admin_tenant = service neutron_admin_username = neutron neutron_admin_password = neutron neutron_admin_auth_url = http://192.168.122.61:35357/v2.0 firewall_driver = nova.virt.firewall.IptablesFirewallDriver security_group_api = neutron compute_driver = nova.virt.libvirt.LibvirtDriver

Note 5: excerpt from api-paste.ini file on the compute host

[filter:authtoken] paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory auth_host = 192.168.122.61 auth_protocol = http auth_port = 35357 admin_user = nova admin_tenant_name = service admin_password = nova

Note 6: output of keystone command from compute host

[root@viserion utils]# keystone endpoint-list +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 | | 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf | | 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 | | d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 | | d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 | | f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac | | fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

[root@viserion nova]# keystone endpoint-get --service network +-------------------+----------------------------+ | Property | Value | +-------------------+----------------------------+ | network.publicURL | http://192.168.122.61:9696 | +-------------------+----------------------------+

==========END COPY/PASTE================

neutron client authentication failed

I am running Havana/Fedora 19 and my nova-compute reports neutron authentication failure. It indicates that the service or the endpoint are not available. I posted a request on the launchpad URL and I got a suggestion to include neutron_region_name in the nova.conf file. I included it on both the controller and the compute nodes and restarted all my services, but I am still running into the same error.

Since I am using RDO packages, I thought I'd post it it on the RDO forum to see if anyone can point me in the right direction. A copy-paste of my question from the launchpad site is attached below.

=========BEGIN COPY/PASTE===========

I am running Havana on a Fedora 19 system and I am running into some issues in nova-compute when booting an instance. I am getting a "neutron client authentication failed: Could not find Service or Region in Service Catalog" error in the nova-compute log file (note 1). However, the endpoints do exist when run from the controller node (note 2). I am also able to use the neutron client to print previously created networks and all other related information (note 3), so I am not quite sure if it's the endpoint that is causing issues. Nova.conf appears to have the right settings to indicate where the neutron is running (note 4) while the api-paste.ini file also the right information (note 5). To verify, I ran keystone commands from the compute host and I am able to see the endpoints (note 6).

It does not appear to be the issue described in https://bugs.launchpad.net/nova/+bug/1241275 since the stack traces are different. From the looks of it, in my case, there is an attempt to get the token and it fails to locate even the endpoint. Any thoughts or suggestions in this regard are greatly appreciated.

Notes:

  1. nova-compute log

log:

2013-11-26 14:02:56.921 2493 ERROR nova.network.neutronv2 [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Neutron client authentication failed: Could not find Service or Region in Service Catalog.
2013-11-26 14:02:56.922 2493 ERROR nova.compute.manager [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Instance failed network setup after 1 attempt(s)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager Traceback (most recent call last):
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager dhcp_options=dhcp_options)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapper
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager res = f(self, context, args, *kwargs)
*args, **kwargs)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.exception(msg, port_id)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 323, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager port_req_body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 392, in _populate_neutron_extension_values
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._refresh_neutron_extensions_cache()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 375, in _refresh_neutron_extensions_cache
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager neutron = neutronv2.get_client(context.get_admin_context())
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 71, in get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager return _get_client(token=token)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 52, in _get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager token = _get_auth_token()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 47, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.error(_('Neutron client authentication failed: %s'), e)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 43, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager httpclient.authenticate()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 245, in authenticate
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._extract_service_catalog(body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 208, in _extract_service_catalog
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager endpoint_type=self.endpoint_type)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 82, in url_for
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager raise exceptions.EndpointNotFound()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager EndpointNotFound: Could not find Service or Region in Service Catalog.

Catalog.

Note 2:

[root@drogon utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

Note 3: Printing net-list as both admin and demo users

[root@viserion utils]# neutron net-list +--------------------------------------+----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+--------------------------------------------------+ | 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 | | eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 7.7.7.0/24 | +--------------------------------------+----------+--------------------------------------------------+

[root@viserion utils]# . ./keystone_demo_remote.bash
[root@viserion utils]# neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 |
| eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 |
+--------------------------------------+----------+--------------------------------------------------+

+--------------------------------------+----------+--------------------------------------------------+

Note 4: excerpt from nova.conf file on compute host

network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.168.122.61:9696
neutron_auth_strategy = keystone
neutron_admin_tenant = service
neutron_admin_username = neutron
neutron_admin_password = neutron
neutron_admin_auth_url = http://192.168.122.61:35357/v2.0
firewall_driver = nova.virt.firewall.IptablesFirewallDriver
security_group_api = neutron
compute_driver = nova.virt.libvirt.LibvirtDriver

nova.virt.libvirt.LibvirtDriver

Note 5: excerpt from api-paste.ini file on the compute host

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 192.168.122.61
auth_protocol = http
auth_port = 35357
admin_user = nova
admin_tenant_name = service
admin_password = nova

nova

Note 6: output of keystone command from compute host

[root@viserion utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ [root@viserion nova]# keystone endpoint-get --service network +-------------------+----------------------------+ | Property | Value | +-------------------+----------------------------+ | network.publicURL | http://192.168.122.61:9696 | +-------------------+----------------------------+

+-------------------+----------------------------+

==========END COPY/PASTE================

neutron client authentication failed

[Edit1: Added new information below based on Dheeru's request]

I am running Havana/Fedora 19 and my nova-compute reports neutron authentication failure. It indicates that the service or the endpoint are not available. I posted a request on the launchpad URL and I got a suggestion to include neutron_region_name in the nova.conf file. I included it on both the controller and the compute nodes and restarted all my services, but I am still running into the same error.

Since I am using RDO packages, I thought I'd post it it on the RDO forum to see if anyone can point me in the right direction. A copy-paste of my question from the launchpad site is attached below.

=========BEGIN COPY/PASTE===========

I am running Havana on a Fedora 19 system and I am running into some issues in nova-compute when booting an instance. I am getting a "neutron client authentication failed: Could not find Service or Region in Service Catalog" error in the nova-compute log file (note 1). However, the endpoints do exist when run from the controller node (note 2). I am also able to use the neutron client to print previously created networks and all other related information (note 3), so I am not quite sure if it's the endpoint that is causing issues. Nova.conf appears to have the right settings to indicate where the neutron is running (note 4) while the api-paste.ini file also the right information (note 5). To verify, I ran keystone commands from the compute host and I am able to see the endpoints (note 6).

It does not appear to be the issue described in https://bugs.launchpad.net/nova/+bug/1241275 since the stack traces are different. From the looks of it, in my case, there is an attempt to get the token and it fails to locate even the endpoint. Any thoughts or suggestions in this regard are greatly appreciated.

Notes:

nova-compute log:

2013-11-26 14:02:56.921 2493 ERROR nova.network.neutronv2 [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Neutron client authentication failed: Could not find Service or Region in Service Catalog.
2013-11-26 14:02:56.922 2493 ERROR nova.compute.manager [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Instance failed network setup after 1 attempt(s)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager Traceback (most recent call last):
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager dhcp_options=dhcp_options)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapper
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager res = f(self, context, *args, **kwargs)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.exception(msg, port_id)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 323, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager port_req_body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 392, in _populate_neutron_extension_values
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._refresh_neutron_extensions_cache()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 375, in _refresh_neutron_extensions_cache
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager neutron = neutronv2.get_client(context.get_admin_context())
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 71, in get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager return _get_client(token=token)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 52, in _get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager token = _get_auth_token()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 47, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.error(_('Neutron client authentication failed: %s'), e)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 43, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager httpclient.authenticate()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 245, in authenticate
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._extract_service_catalog(body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 208, in _extract_service_catalog
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager endpoint_type=self.endpoint_type)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 82, in url_for
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager raise exceptions.EndpointNotFound()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager EndpointNotFound: Could not find Service or Region in Service Catalog.

Note 2:

[root@drogon utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

Note 3: Printing net-list as both admin and demo users

[root@viserion utils]# neutron net-list +--------------------------------------+----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+--------------------------------------------------+ | 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 | | eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 7.7.7.0/24 | +--------------------------------------+----------+--------------------------------------------------+

[root@viserion utils]# . ./keystone_demo_remote.bash
[root@viserion utils]# neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 |
| eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 |
+--------------------------------------+----------+--------------------------------------------------+

Note 4: excerpt from nova.conf file on compute host

network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.168.122.61:9696
neutron_auth_strategy = keystone
neutron_admin_tenant = service
neutron_admin_username = neutron
neutron_admin_password = neutron
neutron_admin_auth_url = http://192.168.122.61:35357/v2.0
firewall_driver = nova.virt.firewall.IptablesFirewallDriver
security_group_api = neutron
compute_driver = nova.virt.libvirt.LibvirtDriver

Note 5: excerpt from api-paste.ini file on the compute host

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 192.168.122.61
auth_protocol = http
auth_port = 35357
admin_user = nova
admin_tenant_name = service
admin_password = nova

Note 6: output of keystone command from compute host

[root@viserion utils]# keystone endpoint-list
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+
| 44e10585f1e44d7f8d77d4ea54594517 | regionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 |
| 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf |
| 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 |
| d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 |
| d68b2d16f6624df3a0893d040a395ea7 | regionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 |
| f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac |
| fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

[root@viserion nova]# keystone endpoint-get --service network
+-------------------+----------------------------+
| Property | Value |
+-------------------+----------------------------+
| network.publicURL | http://192.168.122.61:9696 |
+-------------------+----------------------------+

==========END COPY/PASTE================

Edit 1:

I still see the same issue. Here is an output of the users, tenants and services retrieved from the compute node as suggested by Dheeru.

[root@viserion utils]# cat keystone_remote.bash source variables.env

export OS_USERNAME=admin export OS_PASSWORD=$ADMINPASS export OS_TENANT_NAME=admin export OS_AUTH_URL=http://$OS_CLOUD_CONTROLLER:5000/v2.0

[root@viserion utils]# keystone user-list +----------------------------------+---------+---------+-------+ | id | name | enabled | email | +----------------------------------+---------+---------+-------+ | e53da56b118b47b7a2effcbb84e1fd64 | admin | True | | | 813e6a725f074cd98b2e70821efff038 | cinder | True | | | e41024d64afe4ef5bfe0b23e4ec921cc | demo | True | | | f729ec74d25e421db22b7a8230d015f2 | glance | True | | | 8502991328d54fa78e4fed5123cb3b35 | neutron | True | | | 774b59b1240b4a82844d08b002ccb3f9 | nova | True | | +---------------------------------

[root@viserion]# keystone tenant-list +----------------------------------+---------+---------+ | id | name | enabled | +----------------------------------+---------+---------+ | 781670306e6b4c50bc332e511aca54c4 | admin | True | | a280078745bd482081f7d567f0a6f24f | demo | True | | ef7607ce7f4f4e5b8b1c9533ebf9c11b | service | True | +----------------------------------+---------+---------+ [root@viserion]# keystone service-list +----------------------------------+----------+----------+-------------+ | id | name | type | description | +----------------------------------+----------+----------+-------------+ | 4efbe45a80544d6595e99c4d679cd00b | cinder | volume | Volume V2 | | 090a3ba5f0c54558837de983aff299b6 | cinder2 | volumev2 | Volume V2 | | fc5a5af784764bf9bc2100103eba0121 | ec2 | ec2 | EC2 | | 97ddca39d0d94bb9841ca7fe7914db58 | glance | image | Image | | 41a1f3a32bf74fafafe65dbb588a8085 | keystone | identity | Identity | | 480efc9c07084aefba05f1b61461584e | neutron | network | Network | | 7e902d2371b8441cb2d0683ee1dc2b99 | nova | compute | Compute | +----------------------------------+----------+----------+-------------+

neutron client authentication failed

[Edit1: Added new information below based on Dheeru's request]

[Edit 2: Clarification on Dheeru's second comment]

I am running Havana/Fedora 19 and my nova-compute reports neutron authentication failure. It indicates that the service or the endpoint are not available. I posted a request on the launchpad URL and I got a suggestion to include neutron_region_name in the nova.conf file. I included it on both the controller and the compute nodes and restarted all my services, but I am still running into the same error.

Since I am using RDO packages, I thought I'd post it it on the RDO forum to see if anyone can point me in the right direction. A copy-paste of my question from the launchpad site is attached below.

=========BEGIN COPY/PASTE===========

I am running Havana on a Fedora 19 system and I am running into some issues in nova-compute when booting an instance. I am getting a "neutron client authentication failed: Could not find Service or Region in Service Catalog" error in the nova-compute log file (note 1). However, the endpoints do exist when run from the controller node (note 2). I am also able to use the neutron client to print previously created networks and all other related information (note 3), so I am not quite sure if it's the endpoint that is causing issues. Nova.conf appears to have the right settings to indicate where the neutron is running (note 4) while the api-paste.ini file also the right information (note 5). To verify, I ran keystone commands from the compute host and I am able to see the endpoints (note 6).

It does not appear to be the issue described in https://bugs.launchpad.net/nova/+bug/1241275 since the stack traces are different. From the looks of it, in my case, there is an attempt to get the token and it fails to locate even the endpoint. Any thoughts or suggestions in this regard are greatly appreciated.

Notes:

nova-compute log:

2013-11-26 14:02:56.921 2493 ERROR nova.network.neutronv2 [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Neutron client authentication failed: Could not find Service or Region in Service Catalog.
2013-11-26 14:02:56.922 2493 ERROR nova.compute.manager [req-662e2363-cdcb-495e-96c3-d9852fcfea59 None None] Instance failed network setup after 1 attempt(s)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager Traceback (most recent call last):
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager dhcp_options=dhcp_options)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapper
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager res = f(self, context, *args, **kwargs)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.exception(msg, port_id)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 323, in allocate_for_instance
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager port_req_body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 392, in _populate_neutron_extension_values
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._refresh_neutron_extensions_cache()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 375, in _refresh_neutron_extensions_cache
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager neutron = neutronv2.get_client(context.get_admin_context())
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 71, in get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager return _get_client(token=token)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 52, in _get_client
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager token = _get_auth_token()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 47, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager LOG.error(_('Neutron client authentication failed: %s'), e)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 43, in _get_auth_token
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager httpclient.authenticate()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 245, in authenticate
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager self._extract_service_catalog(body)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 208, in _extract_service_catalog
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager endpoint_type=self.endpoint_type)
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 82, in url_for
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager raise exceptions.EndpointNotFound()
2013-11-26 14:02:56.922 2493 TRACE nova.compute.manager EndpointNotFound: Could not find Service or Region in Service Catalog.

Note 2:

[root@drogon utils]# keystone endpoint-list +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | id | | region | | publicurl | | internalurl | | adminurl | | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | 44e10585f1e44d7f8d77d4ea54594517 | regionOne 25c328d27b06431eae622035b6abbe0d | RegionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 97ddca39d0d94bb9841ca7fe7914db58 | | 26b0381d1f344d7ab6242607ecd50d57 | RegionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 41a1f3a32bf74fafafe65dbb588a8085 | | 28750c9acd8c466b863ac9926ab0f95a | RegionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | fc5a5af784764bf9bc2100103eba0121 | | 74af0b1bedba4397a4f7578b6d7f9d74 | RegionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | 480efc9c07084aefba05f1b61461584e | | 7c22f940793549a7a63af3c0ea89a5de | RegionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | 090a3ba5f0c54558837de983aff299b6 | | a1d31d039f8e4e23843a211d00e539b1 | RegionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 | | 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf | | 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 | | d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 | | d68b2d16f6624df3a0893d040a395ea7 | regionOne 7e902d2371b8441cb2d0683ee1dc2b99 | | c2167d9273da4e03919f840aeb8d67b5 | RegionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 | | f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac | | fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 4efbe45a80544d6595e99c4d679cd00b | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

Note 3: Printing net-list as both admin and demo users

[root@viserion utils]# neutron net-list +--------------------------------------+----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+----------+--------------------------------------------------+ | 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 | | eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 7.7.7.0/24 | +--------------------------------------+----------+--------------------------------------------------+

[root@viserion utils]# . ./keystone_demo_remote.bash
[root@viserion utils]# neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 04676255-320d-4600-84b8-0e69b03928ac | demo-net | 537fe95c-1722-4007-af3e-3c5e91b80592 10.5.5.0/24 |
| eb0ddf1e-0251-4e2d-a7f8-edeef177da30 | ext-net | 86924ec1-84c5-4c20-85fe-05fcb2dbef41 |
+--------------------------------------+----------+--------------------------------------------------+

Note 4: excerpt from nova.conf file on compute host

network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.168.122.61:9696
neutron_auth_strategy = keystone
neutron_admin_tenant = service
neutron_admin_username = neutron
neutron_admin_password = neutron
neutron_admin_auth_url = http://192.168.122.61:35357/v2.0
firewall_driver = nova.virt.firewall.IptablesFirewallDriver
security_group_api = neutron
compute_driver = nova.virt.libvirt.LibvirtDriver

Note 5: excerpt from api-paste.ini file on the compute host

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 192.168.122.61
auth_protocol = http
auth_port = 35357
admin_user = nova
admin_tenant_name = service
admin_password = nova

Note 6: output of keystone command from compute host

[root@viserion utils]# keystone endpoint-list +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | id | | region | | publicurl | | internalurl | | adminurl | | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | 44e10585f1e44d7f8d77d4ea54594517 | regionOne 25c328d27b06431eae622035b6abbe0d | RegionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 97ddca39d0d94bb9841ca7fe7914db58 | | 26b0381d1f344d7ab6242607ecd50d57 | RegionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 41a1f3a32bf74fafafe65dbb588a8085 | | 28750c9acd8c466b863ac9926ab0f95a | RegionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | fc5a5af784764bf9bc2100103eba0121 | | 74af0b1bedba4397a4f7578b6d7f9d74 | RegionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | 480efc9c07084aefba05f1b61461584e | | 7c22f940793549a7a63af3c0ea89a5de | RegionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | 090a3ba5f0c54558837de983aff299b6 | | a1d31d039f8e4e23843a211d00e539b1 | RegionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 37ca06fa667a4148b0532566be4d6e77 | | 7ee8df0d51204c27815fa36e9c95ac1d | regionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 3e7da61c8cd54c45bddc4ce09709c5bf | | 95ecd68ab73643f1b90dc838f01b9d86 | regionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | de825bed415b43fda2ef2af83c51a146 | | d4282404b7674be69dee9143e1d4dcdd | regionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | c18c7e1643f241359b53c281b6609444 | | d68b2d16f6624df3a0893d040a395ea7 | regionOne 7e902d2371b8441cb2d0683ee1dc2b99 | | c2167d9273da4e03919f840aeb8d67b5 | RegionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | f839d5dbbe7d4a30bba17e4ac7b62b22 | | f91f74def62e4ee28900515ae7acf312 | regionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | dab8d63d7341498e8414ae50f74d0aac | | fd7405a77f804cf289a197a52a72b922 | regionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 28fb944cbc154b02950fdccd4b912827 | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ 4efbe45a80544d6595e99c4d679cd00b | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+

[root@viserion nova]# keystone endpoint-get --service network
+-------------------+----------------------------+
| Property | Value |
+-------------------+----------------------------+
| network.publicURL | http://192.168.122.61:9696 |
+-------------------+----------------------------+

==========END COPY/PASTE================

Edit 1:

I still see the same issue. Here is an output of the users, tenants and services retrieved from the compute node as suggested by Dheeru.

[root@viserion utils]# cat keystone_remote.bash source variables.env

export OS_USERNAME=admin export OS_PASSWORD=$ADMINPASS export OS_TENANT_NAME=admin export OS_AUTH_URL=http://$OS_CLOUD_CONTROLLER:5000/v2.0

[root@viserion utils]# keystone user-list +----------------------------------+---------+---------+-------+ | id | name | enabled | email | +----------------------------------+---------+---------+-------+ | e53da56b118b47b7a2effcbb84e1fd64 | admin | True | | | 813e6a725f074cd98b2e70821efff038 | cinder | True | | | e41024d64afe4ef5bfe0b23e4ec921cc | demo | True | | | f729ec74d25e421db22b7a8230d015f2 | glance | True | | | 8502991328d54fa78e4fed5123cb3b35 | neutron | True | | | 774b59b1240b4a82844d08b002ccb3f9 | nova | True | | +---------------------------------

[root@viserion]# keystone tenant-list +----------------------------------+---------+---------+ | id | name | enabled | +----------------------------------+---------+---------+ | 781670306e6b4c50bc332e511aca54c4 | admin | True | | a280078745bd482081f7d567f0a6f24f | demo | True | | ef7607ce7f4f4e5b8b1c9533ebf9c11b | service | True | +----------------------------------+---------+---------+ [root@viserion]# keystone service-list +----------------------------------+----------+----------+-------------+ | id | name | type | description | +----------------------------------+----------+----------+-------------+ | 4efbe45a80544d6595e99c4d679cd00b | cinder | volume | Volume V2 | | 090a3ba5f0c54558837de983aff299b6 | cinder2 | volumev2 | Volume V2 | | fc5a5af784764bf9bc2100103eba0121 | ec2 | ec2 | EC2 | | 97ddca39d0d94bb9841ca7fe7914db58 | glance | image | Image | | 41a1f3a32bf74fafafe65dbb588a8085 | keystone | identity | Identity | | 480efc9c07084aefba05f1b61461584e | neutron | network | Network | | 7e902d2371b8441cb2d0683ee1dc2b99 | nova | compute | Compute | +----------------------------------+----------+----------+-------------+

Edit 2: Apologize for the confusion. I had re-built my setup again from the ground up and hence the mistmatch with the values. I have updated the notes above to reflect the values on the newer settings and the service ids match (as noted blow).

[root@viserion utils]# keystone endpoint-list +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+ | 25c328d27b06431eae622035b6abbe0d | RegionOne | http://192.168.122.61:9292 | http://192.168.122.61:9292 | http://192.168.122.61:9292 | 97ddca39d0d94bb9841ca7fe7914db58 | | 26b0381d1f344d7ab6242607ecd50d57 | RegionOne | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:5000/v2.0 | http://192.168.122.61:35357/v2.0 | 41a1f3a32bf74fafafe65dbb588a8085 | | 28750c9acd8c466b863ac9926ab0f95a | RegionOne | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Cloud | http://192.168.122.61:8773/services/Admin | fc5a5af784764bf9bc2100103eba0121 | | 74af0b1bedba4397a4f7578b6d7f9d74 | RegionOne | http://192.168.122.61:9696 | http://192.168.122.61:9696 | http://192.168.122.61:9696 | 480efc9c07084aefba05f1b61461584e | | 7c22f940793549a7a63af3c0ea89a5de | RegionOne | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | http://192.168.122.61:8776/v2/$(tenant_id)s | 090a3ba5f0c54558837de983aff299b6 | | a1d31d039f8e4e23843a211d00e539b1 | RegionOne | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | http://192.168.122.61:8774/v2/$(tenant_id)s | 7e902d2371b8441cb2d0683ee1dc2b99 | | c2167d9273da4e03919f840aeb8d67b5 | RegionOne | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | http://192.168.122.61:8776/v1/$(tenant_id)s | 4efbe45a80544d6595e99c4d679cd00b | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+---------------------------------------------+----------------------------------+