Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Open vSwitch agent not comming up on second compute using gre tunnel

I have all in one setup running on centos7 and juno. every thing work fine on the all in hosts. I have 2nd compute node ever the compute server is up. but the netutron open vswitch is not coming up.

nova service-list +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-cert | hp3bl1 | internal | enabled | up | 2014-11-26T21:28:37.000000 | - | | 2 | nova-consoleauth | hp3bl1 | internal | enabled | up | 2014-11-26T21:28:37.000000 | - | | 3 | nova-scheduler | hp3bl1 | internal | enabled | up | 2014-11-26T21:28:37.000000 | - | | 4 | nova-conductor | hp3bl1 | internal | enabled | up | 2014-11-26T21:28:43.000000 | - | | 6 | nova-compute | hp3bl1 | nova | enabled | up | 2014-11-26T21:28:43.000000 | - | | 7 | nova-compute | hp3bl2 | nova | enabled | up | 2014-11-26T21:28:41.000000 | - | +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+

neutron agent-list

+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+--------------------+--------+-------+----------------+---------------------------+ | 53cfb703-0681-4b49-b5c2-15f7af93dad8 | Open vSwitch agent | hp3bl1 | :-) | True | neutron-openvswitch-agent | | 8a3b1e6a-3431-40ce-aa77-9519430c63db | L3 agent | hp3bl1 | :-) | True | neutron-l3-agent | | af8dac02-1962-4003-b668-9826d33a0870 | Metadata agent | hp3bl1 | :-) | True | neutron-metadata-agent | | c274d852-f03a-4ea1-afaa-cd8aaf4b1145 | DHCP agent | hp3bl1 | :-) | True | neutron-dhcp-agent | +--------------------------------------+--------------------+--------+-------+----------------+---------------------------+

Open vSwitch agent not comming up on second compute using gre tunnel

I have all in one setup running on centos7 and juno. every thing work fine on the all in hosts. I have 2nd compute node ever the compute server is up. but the netutron open vswitch is not coming up.

 nova service-list
 +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
 | Id | Binary           | Host   | Zone     | Status  | State | Updated_at                 | Disabled Reason |
 +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
 | 1  | nova-cert        | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
 | 2  | nova-consoleauth | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
 | 3  | nova-scheduler   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
 | 4  | nova-conductor   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
 | 6  | nova-compute     | hp3bl1 | nova     | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
 | 7  | nova-compute     | hp3bl2 | nova     | enabled | up    | 2014-11-26T21:28:41.000000 | -               |
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+

+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+


# neutron agent-list 

agent-list +--------------------------------------+--------------------+--------+-------+----------------+---------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+--------------------+--------+-------+----------------+---------------------------+ | 53cfb703-0681-4b49-b5c2-15f7af93dad8 | Open vSwitch agent | hp3bl1 | :-) | True | neutron-openvswitch-agent | | 8a3b1e6a-3431-40ce-aa77-9519430c63db | L3 agent | hp3bl1 | :-) | True | neutron-l3-agent | | af8dac02-1962-4003-b668-9826d33a0870 | Metadata agent | hp3bl1 | :-) | True | neutron-metadata-agent | | c274d852-f03a-4ea1-afaa-cd8aaf4b1145 | DHCP agent | hp3bl1 | :-) | True | neutron-dhcp-agent | +--------------------------------------+--------------------+--------+-------+----------------+---------------------------+

+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+

Open vSwitch agent not comming up on second compute using gre tunnel

I have all in one setup running on centos7 and juno. every thing work fine on the all in hosts. I have 2nd compute node ever the compute server is up. but the netutron open vswitch is not coming up.

   nova service-list
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
    | Id | Binary           | Host   | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
    | 1  | nova-cert        | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 2  | nova-consoleauth | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 3  | nova-scheduler   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 4  | nova-conductor   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
    | 6  | nova-compute     | hp3bl1 | nova     | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
    | 7  | nova-compute     | hp3bl2 | nova     | enabled | up    | 2014-11-26T21:28:41.000000 | -               |
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+

# neutron agent-list
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+
| id                                   | agent_type         | host   | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+
| 53cfb703-0681-4b49-b5c2-15f7af93dad8 | Open vSwitch agent | hp3bl1 | :-)   | True           | neutron-openvswitch-agent |
| 8a3b1e6a-3431-40ce-aa77-9519430c63db | L3 agent           | hp3bl1 | :-)   | True           | neutron-l3-agent          |
| af8dac02-1962-4003-b668-9826d33a0870 | Metadata agent     | hp3bl1 | :-)   | True           | neutron-metadata-agent    |
| c274d852-f03a-4ea1-afaa-cd8aaf4b1145 | DHCP agent         | hp3bl1 | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+

cat openvswitch-agent.log 2014-11-26 21:37:27.014 8704 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file) 2014-11-26 21:37:27.038 8704 ERROR neutron.agent.linux.utils [-] Command: ['ps', '--ppid', '8813', '-o', 'pid='] Exit code: 1 Stdout: '' Stderr: '' 2014-11-26 21:37:27.250 8704 CRITICAL neutron [req-4922694e-4b9e-43df-b887-b3cb2fac728e None] AssertionError: Trying to re-send() an already-triggered event. 2014-11-26 21:37:27.250 8704 TRACE neutron Traceback (most recent call last): 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module> 2014-11-26 21:37:27.250 8704 TRACE neutron sys.exit(main()) 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1545, in main 2014-11-26 21:37:27.250 8704 TRACE neutron agent.daemon_loop() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1472, in daemon_loop 2014-11-26 21:37:27.250 8704 TRACE neutron self.rpc_loop(polling_manager=pm) 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__ 2014-11-26 21:37:27.250 8704 TRACE neutron self.gen.next() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 39, in get_polling_manager 2014-11-26 21:37:27.250 8704 TRACE neutron pm.stop() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 106, in stop 2014-11-26 21:37:27.250 8704 TRACE neutron self._monitor.stop() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 89, in stop 2014-11-26 21:37:27.250 8704 TRACE neutron self._kill() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovsdb_monitor.py", line 99, in _kill 2014-11-26 21:37:27.250 8704 TRACE neutron super(SimpleInterfaceMonitor, self)._kill(args, *kwargs) 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 116, in _kill 2014-11-26 21:37:27.250 8704 TRACE neutron self._kill_event.send() 2014-11-26 21:37:27.250 8704 TRACE neutron File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 155, in send 2014-11-26 21:37:27.250 8704 TRACE neutron assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.' 2014-11-26 21:37:27.250 8704 TRACE neutron AssertionError: Trying to re-send() an already-triggered event. 2014-11-26 21:37:27.250 8704 TRACE neutron 2014-11-26 21:42:01.097 1934 ERROR oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Unable to connect to AMQP server on {'username': '', 'host': 'hp3bl1:5672', 'password': ''}: [ Errno 111] ECONNREFUSED. Sleeping 5 seconds 2014-11-26 21:42:06.107 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.136 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.167 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.195 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.224 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.264 1934 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:06.280 1934 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on hp3bl1:5672 2014-11-26 21:42:07.978 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent initialized successfully, now running... 2014-11-26 21:42:07.984 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent out of sync with plugin! 2014-11-26 21:42:08.063 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent tunnel out of sync with plugin! 2014-11-26 21:43:05.455 1934 INFO neutron.agent.securitygroups_rpc [req-3d51ede4-2633-48eb-a99b-290940697411 None] Provider rule updated 2014-11-26 21:43:06.290 1934 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state! 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 259, in _report_s tate 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.use_call) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 70, in report_state 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return self.call(context, msg) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return method(args, *kwargs) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent context, msg, rpc_method='call', *kwargs) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return func(context, msg['method'], *msg['args']) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return self.prepare().call(ctxt, method, **kwargs) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=self.retry) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent retry=retry) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent reply, ending = self._poll_connection(msg_id, timeout) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent % msg_id) 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 471ce88468374086a77a7d6bffc31e63 2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 2014-11-26 21:43:06.301 1934 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 30.045233 sec 2014-11-26 21:44:06.933 1934 INFO neutron.agent.securitygroups_rpc [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Refresh firewall rules 2014-11-26 21:44:06.934 1934 INFO neutron.agent.securitygroups_rpc [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] No ports here to refresh firewall 2014-11-26 21:44:07.017 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent tunnel out of sync with plugin!

Open vSwitch agent not comming up on second compute using gre tunnel

I have all in one setup running on centos7 and juno. every thing work fine on the all in hosts. I have 2nd compute node ever the compute server is up. but the netutron open vswitch is not coming up.

   nova service-list
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
    | Id | Binary           | Host   | Zone     | Status  | State | Updated_at                 | Disabled Reason |
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
    | 1  | nova-cert        | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 2  | nova-consoleauth | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 3  | nova-scheduler   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:37.000000 | -               |
    | 4  | nova-conductor   | hp3bl1 | internal | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
    | 6  | nova-compute     | hp3bl1 | nova     | enabled | up    | 2014-11-26T21:28:43.000000 | -               |
    | 7  | nova-compute     | hp3bl2 | nova     | enabled | up    | 2014-11-26T21:28:41.000000 | -               |
    +----+------------------+--------+----------+---------+-------+----------------------------+-----------------+

# neutron agent-list
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+
| id                                   | agent_type         | host   | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+
| 53cfb703-0681-4b49-b5c2-15f7af93dad8 | Open vSwitch agent | hp3bl1 | :-)   | True           | neutron-openvswitch-agent |
| 8a3b1e6a-3431-40ce-aa77-9519430c63db | L3 agent           | hp3bl1 | :-)   | True           | neutron-l3-agent          |
| af8dac02-1962-4003-b668-9826d33a0870 | Metadata agent     | hp3bl1 | :-)   | True           | neutron-metadata-agent    |
| c274d852-f03a-4ea1-afaa-cd8aaf4b1145 | DHCP agent         | hp3bl1 | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+--------+-------+----------------+---------------------------+

cat openvswitch-agent.log

2014-11-26 21:37:27.014 8704 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-11-26 21:37:27.038 8704 ERROR neutron.agent.linux.utils [-]
Command: ['ps', '--ppid', '8813', '-o', 'pid=']
Exit code: 1
Stdout: ''
Stderr: ''
2014-11-26 21:37:27.250 8704 CRITICAL neutron [req-4922694e-4b9e-43df-b887-b3cb2fac728e None] AssertionError: Trying to re-send() an already-triggered event.
2014-11-26 21:37:27.250 8704 TRACE neutron Traceback (most recent call last):
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2014-11-26 21:37:27.250 8704 TRACE neutron     sys.exit(main())
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1545, in main
2014-11-26 21:37:27.250 8704 TRACE neutron     agent.daemon_loop()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1472, in daemon_loop
2014-11-26 21:37:27.250 8704 TRACE neutron     self.rpc_loop(polling_manager=pm)
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2014-11-26 21:37:27.250 8704 TRACE neutron     self.gen.next()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 39, in get_polling_manager
2014-11-26 21:37:27.250 8704 TRACE neutron     pm.stop()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/polling.py", line 106, in stop
2014-11-26 21:37:27.250 8704 TRACE neutron     self._monitor.stop()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 89, in stop
2014-11-26 21:37:27.250 8704 TRACE neutron     self._kill()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovsdb_monitor.py", line 99, in _kill
2014-11-26 21:37:27.250 8704 TRACE neutron     super(SimpleInterfaceMonitor, self)._kill(args, *kwargs)
self)._kill(*args, **kwargs)
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 116, in _kill
2014-11-26 21:37:27.250 8704 TRACE neutron     self._kill_event.send()
2014-11-26 21:37:27.250 8704 TRACE neutron   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 155, in send
2014-11-26 21:37:27.250 8704 TRACE neutron     assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2014-11-26 21:37:27.250 8704 TRACE neutron AssertionError: Trying to re-send() an already-triggered event.
2014-11-26 21:37:27.250 8704 TRACE neutron
2014-11-26 21:42:01.097 1934 ERROR oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Unable to connect to AMQP server on {'username': '', 'host': 'hp3bl1:5672', 'password': ''}: [
Errno 111] ECONNREFUSED. Sleeping 5 seconds
2014-11-26 21:42:06.107 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.136 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.167 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.195 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.224 1934 INFO oslo.messaging._drivers.impl_qpid [req-d874e606-8fb6-4ba0-895c-16555d601a30 ] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.264 1934 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:06.280 1934 INFO oslo.messaging._drivers.impl_qpid [-] Connected to AMQP server on hp3bl1:5672
2014-11-26 21:42:07.978 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent initialized successfully, now running...
2014-11-26 21:42:07.984 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent out of sync with plugin!
2014-11-26 21:42:08.063 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent tunnel out of sync with plugin!
2014-11-26 21:43:05.455 1934 INFO neutron.agent.securitygroups_rpc [req-3d51ede4-2633-48eb-a99b-290940697411 None] Provider rule updated
2014-11-26 21:43:06.290 1934 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 259, in _report_s
tate
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.use_call)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 70, in report_state
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return self.call(context, msg)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return method(args, *kwargs)
method(*args, **kwargs)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     context, msg, rpc_method='call', *kwargs)
**kwargs)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in __call_rpc_method
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return func(context, msg['method'], *msg['args'])
**msg['args'])
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in call
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return self.prepare().call(ctxt, method, **kwargs)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retry=retry)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     reply, ending = self._poll_connection(msg_id, timeout)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     % msg_id)
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 471ce88468374086a77a7d6bffc31e63
2014-11-26 21:43:06.290 1934 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
2014-11-26 21:43:06.301 1934 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 30.045233 sec
2014-11-26 21:44:06.933 1934 INFO neutron.agent.securitygroups_rpc [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Refresh firewall rules
2014-11-26 21:44:06.934 1934 INFO neutron.agent.securitygroups_rpc [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] No ports here to refresh firewall
2014-11-26 21:44:07.017 1934 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-d874e606-8fb6-4ba0-895c-16555d601a30 None] Agent tunnel out of sync with plugin!

plugin!