Failed to notify nova on events - Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.

asked 2014-10-02 09:19:25 -0500

marica gravatar image

Dear all,

we are experiencing some problems with our openstack infrastructure after the upgrade from havana to icehouse (on Ubuntu 12.04).

We have successfully upgraded the controller and network nodes (the neutron server is running on the network node) with quite no downtime using the upgrade compatibility flag set to icehouse-compat in nova.conf.

All the existing VMs running on the **havana compute nodes** were not affected by the upgrade and worked normally. Note that we are using neutron + ml2 plugin with ovs configured for both flat and gre networks. Now, we are encountering problems with the gre network starting new VM on the **havana compute nodes** whereas the public flat network works fine.

Moreover we are also experiencing some problems as we started to upgrade the compute nodes: we have selected one compute node and executed dist-upgrade using the icehouse repo. Unfortunately, something goes wrong in the VM instatiation on this **icehouse** compute node and it seems to be related to the external event notification between the neutron server and the nova service /os-server-external-events. In more details, we get the following:

  • on the compute node, a timeout is notified during the instance creation:
WARNING nova.virt.libvirt.driver [req-a1e6e66e-9b18-458a-b31e-0d64eb098e81 769b7fafd6c145ccb8680e398702f14f 57f427029f1a45cc862a0c1321fd5ff6] Timeout waiting for vif plugging callback for instance 65ffc677-0bd4-4ef6-84ef-852fad012fd1 

We had to add the following two parameters in the nova.conf file to carry out successfully the VM instantiation:

vif_plugging_is_fatal = False
vif_plugging_timeout = 2
  • On the network node we get (/var/log/neutron/server.log):
2014-10-02 15:38:50.232 4559 DEBUG urllib3.connectionpool [-] "POST /v2/15ee65175a6a4548b5422bb276732580/os-server-external-events HTTP/1.1" 500 128 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-10-02 15:38:50.234 4559 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'3d06fab7-5b82-4122-b519-6bed39b78ed9', 'name': 'network-vif-plugged', 'server_uuid': u'65ffc677-0bd4-4ef6-84ef-852fad012fd1'}]
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova Traceback (most recent call last):
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 222, in send_events
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova     batched_events)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova     return_raw=True)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova     _resp, body = self.api.client.post(url, body=body)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova     return self._cs_request(url, 'POST', **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
2014-10-02 15:38:50.234 4559 TRACE neutron ...
(more)
edit retag flag offensive close merge delete