Ask Your Question
2

New Instance gets always into error status [closed]

asked 2013-11-20 06:49:44 -0600

kemra102 gravatar image

updated 2013-11-22 10:40:46 -0600

dheeru gravatar image

When trying to create a nova instance I get no error but after running nova list it shows in an ERROR status:

+--------------------------------------+---------------+--------+------------+-------------+----------+
| ID                                   | Name          | Status | Task State | Power State | Networks |
+--------------------------------------+---------------+--------+------------+-------------+----------+
| f5c25384-5bae-47ed-93e5-6e4cdd55c4a9 | myfirstserver | ERROR  | None       | NOSTATE     |          |
+--------------------------------------+---------------+--------+------------+-------------+----------+

The only errors I can find logged are in /var/log/glance/api.log:

2013-11-20 12:39:33.633 11918 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.3.16
2013-11-20 12:39:34.639 11918 WARNING keystoneclient.middleware.auth_token [-] Unable to find authentication token in headers
2013-11-20 12:39:34.639 11918 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

I have checked the keystone details in all the relevant places but can see nothing amiss.

Edited to provide requested info:

keystone token-get returns a token as expected, no errors.

keystone user-list:

+----------------------------------+--------+---------+----------------+
|                id                |  name  | enabled |     email      |
+----------------------------------+--------+---------+----------------+
| 5fc621f9f24941febb883f78d2d350ec | admin  |   True  | root@localhost |
| f13b1b93cf6f41faaafb0d4be453109a | cinder |   True  | root@localhost |
| b2d3f462387442cfaaad51490d4146f4 | glance |   True  | root@localhost |
| ed506136092e48ceacc3137e1c63f38d |  nova  |   True  | root@localhost |
+----------------------------------+--------+---------+----------------+

keystone tenant-list:

+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| c22af115b6a84dc78b1ae19385c6cd23 |  admin  |   True  |
| 15b85349d46a4bddb48cac7c64bce97d | service |   True  |
+----------------------------------+---------+---------+

keystone role-list:

+----------------------------------+----------+
|                id                |   name   |
+----------------------------------+----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9764c960e12e4cfea424ba2c808d763e |  admin   |
+----------------------------------+----------+

glance image-list:

+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| ID                                   | Name            | Disk Format | Container Format | Size     | Status |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | qcow2       | bare             | 13147648 | active |
+--------------------------------------+-----------------+-------------+------------------+----------+--------+

nova image-list:

+--------------------------------------+-----------------+--------+--------+
| ID                                   | Name            | Status | Server |
+--------------------------------------+-----------------+--------+--------+
| 64640ca4-0c6d-4682-b05b-d7d2976ee02e | My CirrOS Image | ACTIVE |        |
+--------------------------------------+-----------------+--------+--------+

echo $OS_USERNAME && echo $OS_TENANT_NAME && echo $OS_AUTH_URL && echo $OS_PASSWORD
admin
admin
http://192.168.3.16:35357/v2.0
<mypassword>

Here's a pastebin of the command with the debug option turned on: http://pastebin.com/qKZVWE1n

Additional output found in /var/log/glance/registry.log:

2013-11-21 07:16:54.425 1975 INFO keystoneclient.middleware.auth_token [-] Starting keystone auth_token middleware
2013-11-21 07:16:54.537 1975 WARNING keystoneclient.middleware.auth_token [-] Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint

Server is an Admin/Controller node. There are 6 additional Compute nodes that run Nova Compute and Nova Network in multi-home mode. All of these appear to be configured correctly.

I have added admin_token to /etc/nova/api-paste.ini on the a controller and all compute nodes, restarted the services but this makes no difference other than the error in /var/log/glance/registry.log has disappeared.

Output from one of the compute nodes in /var/log/libvirt/libvirtd.log:

2013-11-20 15:35:09.487+0000: 1880: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-20 15:35:09.487+0000: 1880: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 07:38:28.283+0000: 1819: info : libvirt version: 0.10.2, package: 18.el6_4.15 (CentOS BuildSystem <http://bugs.centos.org>, 2013-11-13-10:44:00, c6b8.bsys.dev.centos.org)
2013-11-21 07:38:28.283+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 08:30:00.082+0000: 1819: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
2013-11-21 13:53:15.268+0000: 1827 ...
(more)
edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by dheeru
close date 2013-11-22 20:42:30.288567

3 answers

Sort by ยป oldest newest most voted
1

answered 2013-11-22 03:41:59 -0600

kemra102 gravatar image

After starting some more instances and reading logs across multiple compute nodes this little gem showed up in /var/log/nova/compute.log:

TRACE nova.openstack.common.rpc.amqp ImageNotAuthorized: Not authorized for image

Turns out I needed to add the following to /etc/nova/nova.conf (which was not in the OpenStack docs I used to build the environment):

auth_strategy=keystone

Then I just restarted nova-network & nova-compute across the Compute nodes and it all started working!

edit flag offensive delete link more

Comments

That is cool. Hope some inputs helped you. If you think you got some direction, you can vote my ans link. It is there in there in the following link. [DEFAULT] ... auth_strategy=keystone ... [database] http://docs.openstack.org/havana/install-guide/install/apt/content/nova-compute.html looks like you missed it. This is great new now.. Now can we close the case ?

dheeru gravatar imagedheeru ( 2013-11-22 05:24:00 -0600 )edit

Since your question is cleared can we close the case ?

dheeru gravatar imagedheeru ( 2013-11-22 08:19:56 -0600 )edit
0

answered 2013-11-20 07:25:58 -0600

dheeru gravatar image

Clearly token is missing. Can you try with following command to check that other things are working fine.

  1. Are you able to run 'keystone token-get' , 'keystone user-list', 'keystone tenant-list', 'keystone role-list'
  2. Are you able to run the 'glance image-list'
  3. Are you able to run 'nova image-list'.
  4. What are the values set for followin env variables ? export OS_USERNAME=admin export OS_TENANT_NAME=admin export OS_AUTH_URL=http://192.168.72.100:35357/v2.0 export OS_PASSWORD=welcome123
  5. Can you run your command with --debug option.

You can look /var/log/nova/api*.log

edit flag offensive delete link more

Comments

Updated my original post. The nova logs show nothing at all except a warning that the instance has been set to ERROR state.

kemra102 gravatar imagekemra102 ( 2013-11-20 08:52:50 -0600 )edit

You have issue with creating the instance. Error you pasted may not be related to this. Is it a multi-node setup ? If yes, can ensure that all the services are running perfectly i.e compute, network etc. Are you using the nova-network ? If yes, ensure that service is running.

dheeru gravatar imagedheeru ( 2013-11-20 10:20:43 -0600 )edit

Updated question. Using multiple nodes with compute and nova-network as advised above.

kemra102 gravatar imagekemra102 ( 2013-11-21 01:49:52 -0600 )edit

Since you are using the multi node setup, there must be multiple compute nodes. So request to instantiate the instance can potentailly go to different compute nodes. There may be a issue with hypervisor etc. Can see or send me the following logs from all the compute nodes ? /var/log/libvirt/libvirtd.log /var/log/nova/nova-compute.log Note - How to find multiple compute nodes ? run the command 'nova host-list'. Look for entries with service as 'compute'.

dheeru gravatar imagedheeru ( 2013-11-21 02:40:32 -0600 )edit

Provided a libvirtd.log from one of the compute servers above (it's the entire log).

kemra102 gravatar imagekemra102 ( 2013-11-21 08:26:38 -0600 )edit
0

answered 2013-11-21 06:09:59 -0600

Srikanth Mergu gravatar image

updated 2013-11-21 06:11:52 -0600

Hi,

Add admin_token in nova/api-paste.ini from keystone/keystone.conf

For example: [filter:authtoken] paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory auth_host=IP auth_uri=http://IP:5000 admin_tenant_name=service admin_user=nova admin_password=password admin_token=2fbab144200f45df1f41 (replace this with your admin token)

edit flag offensive delete link more

Comments

That didn't seem to help, see my edited post.

kemra102 gravatar imagekemra102 ( 2013-11-21 08:25:53 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-11-20 06:49:44 -0600

Seen: 14,378 times

Last updated: Nov 22 '13