Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Invalid user token - rejecting request

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Invalid user token - rejecting request

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Edited to provide requested info:

keystone token-get returns a token as expected, no errors.

keystone user-list:

+----------------------------------+--------+---------+----------------+
|                id                |  name  | enabled |     email      |
+----------------------------------+--------+---------+----------------+
| 5fc621f9f24941febb883f78d2d350ec | admin  |   True  | root@localhost |
| f13b1b93cf6f41faaafb0d4be453109a | cinder |   True  | root@localhost |
| b2d3f462387442cfaaad51490d4146f4 | glance |   True  | root@localhost |
| ed506136092e48ceacc3137e1c63f38d |  nova  |   True  | root@localhost |
+----------------------------------+--------+---------+----------------+

keystone tenant-list:

+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| c22af115b6a84dc78b1ae19385c6cd23 |  admin  |   True  |
| 15b85349d46a4bddb48cac7c64bce97d | service |   True  |
+----------------------------------+---------+---------+

keystone role-list:

+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9764c960e12e4cfea424ba2c808d763e |  admin   |
+----------------------------------+----------+

glance image-list:

+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | qcow2       | bare             | 13147648 | active |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+

nova image-list:

+--------------------------------------+-----------------+--------+--------+
| ID                                   | Name            | Status | Server |
+--------------------------------------+-----------------+--------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | ACTIVE |        |
+--------------------------------------+-----------------+--------+--------+

echo $OS_USERNAME && echo $OS_TENANT_NAME && echo $OS_AUTH_URL && echo $OS_PASSWORD
admin
admin
http://192.168.3.16:35357/v2.0
<mypassword>

Here's a pastebin of the command with the debug option turned on: http://pastebin.com/qKZVWE1n

Invalid user token - rejecting request

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Edited to provide requested info:

keystone token-get returns a token as expected, no errors.

keystone user-list:

+----------------------------------+--------+---------+----------------+
|                id                |  name  | enabled |     email      |
+----------------------------------+--------+---------+----------------+
| 5fc621f9f24941febb883f78d2d350ec | admin  |   True  | root@localhost |
| f13b1b93cf6f41faaafb0d4be453109a | cinder |   True  | root@localhost |
| b2d3f462387442cfaaad51490d4146f4 | glance |   True  | root@localhost |
| ed506136092e48ceacc3137e1c63f38d |  nova  |   True  | root@localhost |
+----------------------------------+--------+---------+----------------+

keystone tenant-list:

+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| c22af115b6a84dc78b1ae19385c6cd23 |  admin  |   True  |
| 15b85349d46a4bddb48cac7c64bce97d | service |   True  |
+----------------------------------+---------+---------+

keystone role-list:

+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9764c960e12e4cfea424ba2c808d763e |  admin   |
+----------------------------------+----------+

glance image-list:

+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | qcow2       | bare             | 13147648 | active |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+

nova image-list:

+--------------------------------------+-----------------+--------+--------+
| ID                                   | Name            | Status | Server |
+--------------------------------------+-----------------+--------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | ACTIVE |        |
+--------------------------------------+-----------------+--------+--------+

echo $OS_USERNAME && echo $OS_TENANT_NAME && echo $OS_AUTH_URL && echo $OS_PASSWORD
admin
admin
http://192.168.3.16:35357/v2.0
<mypassword>

Here's a pastebin of the command with the debug option turned on: http://pastebin.com/qKZVWE1n

Additional output found in /var/log/glance/registry.log:

2013-11-21 07:16:54.425 1975 INFO keystoneclient.middleware.auth_token [-] Starting keystone auth_token middleware
2013-11-21 07:16:54.537 1975 WARNING keystoneclient.middleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint

Server is an Admin/Controller node. There are 6 additional Compute nodes that run Nova Compute and Nova Network in multi-home mode. All of these appear to be configured correctly.

Invalid user token - rejecting request

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Edited to provide requested info:

keystone token-get returns a token as expected, no errors.

keystone user-list:

+----------------------------------+--------+---------+----------------+
|                id                |  name  | enabled |     email      |
+----------------------------------+--------+---------+----------------+
| 5fc621f9f24941febb883f78d2d350ec | admin  |   True  | root@localhost |
| f13b1b93cf6f41faaafb0d4be453109a | cinder |   True  | root@localhost |
| b2d3f462387442cfaaad51490d4146f4 | glance |   True  | root@localhost |
| ed506136092e48ceacc3137e1c63f38d |  nova  |   True  | root@localhost |
+----------------------------------+--------+---------+----------------+

keystone tenant-list:

+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| c22af115b6a84dc78b1ae19385c6cd23 |  admin  |   True  |
| 15b85349d46a4bddb48cac7c64bce97d | service |   True  |
+----------------------------------+---------+---------+

keystone role-list:

+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9764c960e12e4cfea424ba2c808d763e |  admin   |
+----------------------------------+----------+

glance image-list:

+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | qcow2       | bare             | 13147648 | active |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+

nova image-list:

+--------------------------------------+-----------------+--------+--------+
| ID                                   | Name            | Status | Server |
+--------------------------------------+-----------------+--------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | ACTIVE |        |
+--------------------------------------+-----------------+--------+--------+

echo $OS_USERNAME && echo $OS_TENANT_NAME && echo $OS_AUTH_URL && echo $OS_PASSWORD
admin
admin
http://192.168.3.16:35357/v2.0
<mypassword>

Here's a pastebin of the command with the debug option turned on: http://pastebin.com/qKZVWE1n

Additional output found in /var/log/glance/registry.log:

2013-11-21 07:16:54.425 1975 INFO keystoneclient.middleware.auth_token [-] Starting keystone auth_token middleware
2013-11-21 07:16:54.537 1975 WARNING keystoneclient.middleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint

Server is an Admin/Controller node. There are 6 additional Compute nodes that run Nova Compute and Nova Network in multi-home mode. All of these appear to be configured correctly.

I have added admin_token to /etc/nova/api-paste.ini on the a controller and all compute nodes, restarted the services but this makes no difference other than the error in /var/log/glance/registry.log has disappeared.

Output from one of the compute nodes in /var/log/libvirt/libvirtd.log:

2013-11-20 15:35:09.487+0000: 1880: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-20 15:35:09.487+0000: 1880: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 07:38:28.283+0000: 1819: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-21 07:38:28.283+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 08:30:00.082+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 13:53:15.268+0000: 1827: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-21 13:53:15.268+0000: 1827: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error

Nova logs show only errors relating to the compute nodes waiting for my controller to boot up this morning when i turned the servers on. There has been nothing since then today.

click to hide/show revision 5
No.5 Revision

Invalid user token - rejecting requestNew Instance gets always into error status

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Edited to provide requested info:

keystone token-get returns a token as expected, no errors.

keystone user-list:

+----------------------------------+--------+---------+----------------+
|                id                |  name  | enabled |     email      |
+----------------------------------+--------+---------+----------------+
| 5fc621f9f24941febb883f78d2d350ec | admin  |   True  | root@localhost |
| f13b1b93cf6f41faaafb0d4be453109a | cinder |   True  | root@localhost |
| b2d3f462387442cfaaad51490d4146f4 | glance |   True  | root@localhost |
| ed506136092e48ceacc3137e1c63f38d |  nova  |   True  | root@localhost |
+----------------------------------+--------+---------+----------------+

keystone tenant-list:

+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| c22af115b6a84dc78b1ae19385c6cd23 |  admin  |   True  |
| 15b85349d46a4bddb48cac7c64bce97d | service |   True  |
+----------------------------------+---------+---------+

keystone role-list:

+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9764c960e12e4cfea424ba2c808d763e |  admin   |
+----------------------------------+----------+

glance image-list:

+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | qcow2       | bare             | 13147648 | active |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+

nova image-list:

+--------------------------------------+-----------------+--------+--------+
| ID                                   | Name            | Status | Server |
+--------------------------------------+-----------------+--------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | ACTIVE |        |
+--------------------------------------+-----------------+--------+--------+

echo $OS_USERNAME && echo $OS_TENANT_NAME && echo $OS_AUTH_URL && echo $OS_PASSWORD
admin
admin
http://192.168.3.16:35357/v2.0
<mypassword>

Here's a pastebin of the command with the debug option turned on: http://pastebin.com/qKZVWE1n

Additional output found in /var/log/glance/registry.log:

2013-11-21 07:16:54.425 1975 INFO keystoneclient.middleware.auth_token [-] Starting keystone auth_token middleware
2013-11-21 07:16:54.537 1975 WARNING keystoneclient.middleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint

Server is an Admin/Controller node. There are 6 additional Compute nodes that run Nova Compute and Nova Network in multi-home mode. All of these appear to be configured correctly.

I have added admin_token to /etc/nova/api-paste.ini on the a controller and all compute nodes, restarted the services but this makes no difference other than the error in /var/log/glance/registry.log has disappeared.

Output from one of the compute nodes in /var/log/libvirt/libvirtd.log:

2013-11-20 15:35:09.487+0000: 1880: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-20 15:35:09.487+0000: 1880: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 07:38:28.283+0000: 1819: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-21 07:38:28.283+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 08:30:00.082+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 13:53:15.268+0000: 1827: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-21 13:53:15.268+0000: 1827: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error

Nova logs show only errors relating to the compute nodes waiting for my controller to boot up this morning when i turned the servers on. There has been nothing since then today.