Ask Your Question
0

Mitaka : Instance in error state due to Failed to notify nova on events

asked 2016-08-12 08:42:48 -0500

Michel gravatar image

Hi,

When launching cirros instance, the compute is waiting for vif plugging callback from the controller. Unfortunately neutron controller is not able to send the event and then compute remove the instance (timeout).

This is a fresh install of Mitaka (no upgrade from older version) No issue with Kilo on the same environment (3-nodes).

Any help would be highly appreciated

Thanks & Regards,

Michel

Controller : VM - CentOs 7 Compute : G8 CentOs 7

SELinux : permissive on both nodes Firewalld disabled on both nodes

From controller neutron server.log

2016-08-12 14:51:30.067 3070 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'6a99763c-8814-4659-9ae5-d4db6c50f7eb', 'name': 'network-vif-plugged', 'server_uuid': u'433be2f0-91a2-4a23-a9ee-e118c228ab62'}]

2016-08-12 14:51:22.370 3068 INFO neutron.wsgi [req-3a96bcca-05c2-4955-9972-4c47bead52ee 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.40 - - [12/Aug/2016 14:51:22] "GET /v2.0/networks/bf33523d-589c-40d9-b465-b0c5dfd708ae HTTP/1.1" 200 877 0.076181
2016-08-12 14:51:22.528 3068 INFO neutron.wsgi [req-a7152096-9516-4e25-b203-4b7391c30992 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.40 - - [12/Aug/2016 14:51:22] "GET /v2.0/networks.json?id=bf33523d-589c-40d9-b465-b0c5dfd708ae HTTP/1.1" 200 880 0.074319
2016-08-12 14:51:22.539 3068 INFO neutron.wsgi [req-7e755bc6-cca2-483b-9888-7b89c2e6ec8e 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.40 - - [12/Aug/2016 14:51:22] "GET /v2.0/quotas/9004b5f9eb7a40b4906d7e197abf6903.json HTTP/1.1" 200 385 0.007345
2016-08-12 14:51:22.604 3068 INFO neutron.wsgi [req-1a3fd2e2-3169-45cf-9e72-1b000aea6803 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.40 - - [12/Aug/2016 14:51:22] "GET /v2.0/ports.json?fields=id&tenant_id=9004b5f9eb7a40b4906d7e197abf6903 HTTP/1.1" 200 369 0.061353
2016-08-12 14:51:23.311 3068 INFO neutron.wsgi [req-8831b616-2a4d-41f7-a8f9-72f863822bf7 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.40 - - [12/Aug/2016 14:51:23] "GET /v2.0/ports.json?device_id=433be2f0-91a2-4a23-a9ee-e118c228ab62 HTTP/1.1" 200 226 0.070872
2016-08-12 14:51:23.993 3068 INFO neutron.wsgi [req-5564fca4-13fc-4821-9c27-2fe58a181062 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.200 - - [12/Aug/2016 14:51:23] "GET /v2.0/extensions.json HTTP/1.1" 200 6007 0.005699
2016-08-12 14:51:24.128 3068 INFO neutron.wsgi [req-bfd0e2da-3ae0-4927-8202-1547ba63c891 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.200 - - [12/Aug/2016 14:51:24] "GET /v2.0/networks.json?id=bf33523d-589c-40d9-b465-b0c5dfd708ae HTTP/1.1" 200 880 0.127760
2016-08-12 14:51:24.240 3068 INFO neutron.wsgi [req-28ed606c-e091-4b7e-b1cd-8bbf72da51da 40ea01e3ad834163b400129f6dae4b15 9004b5f9eb7a40b4906d7e197abf6903 - - -] 10.0.0.200 - - [12/Aug/2016 14:51:24] "GET /v2.0/security-groups.json?tenant_id=9004b5f9eb7a40b4906d7e197abf6903 HTTP/1.1" 200 2532 0.099570
2016-08-12 14:51:25.451 3068 INFO neutron.wsgi [req-1b74a43d-9d85-43cd-a13f-d72d3f116296 1a2e4525db5448d8a9985df7b79090c9 273f07c1bcad4ae5a8c607286cccc37a - - -] 10.0.0.200 - - [12/Aug/2016 14:51:25] "POST /v2.0/ports.json HTTP/1.1" 201 1071 0.877761
2016-08-12 14:51:25.536 3068 INFO neutron.wsgi [req-23a0aa5a-a7c1-4d27-b8ab-8a3f884f542c 1a2e4525db5448d8a9985df7b79090c9 273f07c1bcad4ae5a8c607286cccc37a - - -] 10.0.0.200 - - [12/Aug/2016 14:51:25] "GET /v2.0/ports.json?tenant_id=9004b5f9eb7a40b4906d7e197abf6903&device_id=433be2f0-91a2-4a23-a9ee-e118c228ab62 HTTP/1.1" 200 1069 0.049650
2016-08-12 14:51:25.559 3068 INFO neutron.wsgi [req-4bd3bc97-5a1e-41c8-a648-39fca9118c7a 1a2e4525db5448d8a9985df7b79090c9 273f07c1bcad4ae5a8c607286cccc37a - - -] 10.0.0.200 - - [12/Aug/2016 14:51:25] "GET /v2.0/floatingips.json?fixed_ip_address=212.166.53 ...
(more)
edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2016-08-12 09:31:43 -0500

Michel gravatar image

Hi Barry,

Thanks for your answer !

here are my demo and admin files :

[root@controller openstack]# cat /usr/openstack/admin-openrc
export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=ADMIN_PSW
export OS_AUTH_URL=http://controller:35357/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

[root@controller openstack]# cat /usr/openstack/demo-openrc
export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=demo
export OS_USERNAME=demo
export OS_PASSWORD=DEMO_PSW
export OS_AUTH_URL=http://controller:5000/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2

I am able to request tokens but I am still facing the same issue...

[root@controller openstack]# . admin-openrc
[root@controller openstack]# openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2016-08-12T14:59:17.995362Z                                                                                                                                                             |
| id         | gAAAAABXrdY1e92hOP-XbMptyTNP8qqlWGSmwrOyUn7R0i-XJ8qXNURiUrX5v5r0toUegpH4du3S4g7kA-dFEStkX_7ofddwH_mnsvXg7De1rZaQFe9NK3iRjjwLy-Yvku9AI5l9tshaQIus32AeTFPrQLtlo9jsbvF3DxR5j-ED_jBoQo157is |
| project_id | 9004b5f9eb7a40b4906d7e197abf6903                                                                                                                                                        |
| user_id    | 40ea01e3ad834163b400129f6dae4b15                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@controller openstack]# . demo-openrc
[root@controller openstack]# openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2016-08-12T14:59:44.458172Z                                                                                                                                                             |
| id         | gAAAAABXrdZQGPTq2RecbgcPiQOptO3X1XiIp3Vj7_EaMHCvk2Pe_mQcFKLAUQ-mghF-3pvaiBiEUIx3TWykJkrkD0Nn5LBKtx-nq9tVeH1LJnn1bYM-lIkU1q7-LTtKH3x_0HqP7PzC65CNo2YbTagXYWwB-_0-dccfsV4SNsMTVxee73yh1qw |
| project_id | d1aa453f237041239787fa046a734c11                                                                                                                                                        |
| user_id    | f198c46485bb4de68f99a359c86483e6                                                                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
edit flag offensive delete link more

Comments

Check your /etc/neutron/neutron.conf configuration, specifically the [keystone_authtoken] section against the documentation. Also check the configuration of /etc/nova/nova.conf specifically the [neutron] section. You will find these on both the controller and the compute nodes.

barry.davis gravatar imagebarry.davis ( 2016-08-12 10:28:25 -0500 )edit
1

It's fixed now... An entry was missing in /etc/neutron/neutron.conf Many thanks for your help

Michel gravatar imageMichel ( 2016-08-16 07:16:53 -0500 )edit

@Michel which entry? :)

bertly gravatar imagebertly ( 2016-08-25 04:28:13 -0500 )edit
0

answered 2016-08-12 08:55:24 -0500

As stated in the keystone log, you are not authorized to make the request. Assuming you have a project named demo and a user named demo, you should create a file containing the environment variables needed for authentication. Something like demo-openrc:

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=demo
export OS_USERNAME=demo
export OS_PASSWORD=DEMO_PASSWORD
export OS_AUTH_URL=http://CONTROLLER:5000/v3
export OS_IDENTITY_API_VERSION=3
export OS_IMAGE_API_VERSION=2
PS1='[\u@\h \W]\$ (demo) '

where DEMO_PASSWORD is the password assigned to the demo user, and CONTROLLER is the name of your controller. Source the file with . demo-openrc You should then be able to confirm authentication using openstack token issue If that is successful, then try to create the instance again.

See this document for reference.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-08-12 08:42:48 -0500

Seen: 416 times

Last updated: Aug 12 '16