Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

DHCP and Instances Addressing stopped working

Hi,

I'm running openStack with the following Setup

  • 1x Controller Node has ( Glance, Neutron, Horizon, MySQl, Nova ) and the network model is GRE "CONFIG_NOVA_NETWORK_MANAGER=nova.network.manager.FlatDHCPManager"
  • 1x Compute Node ( nova-compute )

OS : CentOS 6 - 64bit + KVM

The setup was completely stable until today, Suddenly all new vm's can't get new IP from the neutron server and i have old vm's currently running if one of them is restarted it will not get the local IP Address. So i think there may be a problem withing dhcp or any other stuff.

From Controller Node

~ cat /var/log/neutron/dhcp-agent.log

2014-07-19 01:18:30.322 11403 INFO neutron.openstack.common.service [-] Caught SIGTERM, exiting
2014-07-19 01:18:30.867 30198 INFO neutron.common.config [-] Logging enabled!
2014-07-19 01:18:30.868 30198 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2014-07-19 01:18:30.882 30198 INFO neutron.agent.linux.dhcp [-] dnsmasq version: 2.48
2014-07-19 01:18:30.998 30198 INFO neutron.agent.dhcp_agent [-] DHCP agent started
2014-07-19 01:18:31.027 30198 INFO neutron.agent.dhcp_agent [-] Synchronizing state

From my Compute Node

~ tail -n 100 /var/log/neutron/openvswitch-agent.log

    2014-07-19 01:22:49.717 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:09.818 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:10.470 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:10.471 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:10.471 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:30.588 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:31.177 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:31.179 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:31.179 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:51.303 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:51.872 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:51.873 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:51.874 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!

Installed Packages and Versions.

~ rpm -qa | grep -i openstack - On control Node
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-puppet-modules-2013.2-9.1.el6.noarch
openstack-ceilometer-api-2013.2.3-2.el6.noarch
openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
openstack-nova-scheduler-2013.2.3-1.el6.noarch
openstack-ceilometer-common-2013.2.3-2.el6.noarch
python-django-openstack-auth-1.1.2-1.el6.noarch
openstack-ceilometer-central-2013.2.3-2.el6.noarch
openstack-ceilometer-collector-2013.2.3-2.el6.noarch
openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
openstack-nova-common-2013.2.3-1.el6.noarch
openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
openstack-glance-2013.2.3-2.el6.noarch
openstack-nova-conductor-2013.2.3-1.el6.noarch
openstack-nova-novncproxy-2013.2.3-1.el6.noarch
openstack-nova-cert-2013.2.3-1.el6.noarch
openstack-keystone-2013.2.3-3.el6.noarch
openstack-neutron-2013.2.3-9.el6.noarch
openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
openstack-dashboard-2013.2.3-1.el6.noarch
openstack-nova-api-2013.2.3-1.el6.noarch
openstack-nova-console-2013.2.3-1.el6.noarch
openstack-utils-2013.2-2.el6.noarch

~ rpm -qa | grep -i openstack - On Compute Node
openstack-ceilometer-common-2013.2.3-2.el6.noarch
openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
openstack-utils-2013.2-2.el6.noarch
openstack-ceilometer-compute-2013.2.3-2.el6.noarch
openstack-nova-compute-2013.2.3-1.el6.noarch
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-nova-common-2013.2.3-1.el6.noarch
openstack-neutron-2013.2.3-9.el6.noarch

dhcp-agent is running on the controller node.

neutron  30198  0.4  0.1 275060 33732 ?        S    01:18   0:03 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

Help please.

DHCP and Instances Addressing stopped working

Hi,

I'm running openStack with the following Setup

  • 1x Controller Node has ( Glance, Neutron, Horizon, MySQl, Nova ) and the network model is GRE "CONFIG_NOVA_NETWORK_MANAGER=nova.network.manager.FlatDHCPManager"
  • 1x Compute Node ( nova-compute )

OS : CentOS 6 - 64bit + KVM

The setup was completely stable until today, Suddenly all new vm's can't get new IP from the neutron server and i have old vm's currently running if one of them is restarted it will not get the local IP Address. So i think there may be a problem withing dhcp or any other stuff.

From Controller Node

~ cat /var/log/neutron/dhcp-agent.log

2014-07-19 01:18:30.322 11403 INFO neutron.openstack.common.service [-] Caught SIGTERM, exiting
2014-07-19 01:18:30.867 30198 INFO neutron.common.config [-] Logging enabled!
2014-07-19 01:18:30.868 30198 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2014-07-19 01:18:30.882 30198 INFO neutron.agent.linux.dhcp [-] dnsmasq version: 2.48
2014-07-19 01:18:30.998 30198 INFO neutron.agent.dhcp_agent [-] DHCP agent started
2014-07-19 01:18:31.027 30198 INFO neutron.agent.dhcp_agent [-] Synchronizing state

From my Compute Node

~ tail -n 100 /var/log/neutron/openvswitch-agent.log

    2014-07-19 01:22:49.717 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:09.818 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:10.470 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:10.470 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:10.471 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:10.471 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:30.588 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:31.177 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:31.177 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:31.179 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:31.179 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-19 01:23:51.303 2444 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
2014-07-19 01:23:51.872 2444 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1065, in rpc_loop
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1000, in process_network_ports
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 901, in treat_devices_added
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 371, in _apply
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables = self.execute(args, root_helper=self.root_helper)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no filter matched)\n'
2014-07-19 01:23:51.872 2444 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-19 01:23:51.873 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-19 01:23:51.874 2444 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!

http://paste.openstack.org/show/87197/

Installed Packages and Versions.

~ rpm -qa | grep -i openstack - On control Node
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-puppet-modules-2013.2-9.1.el6.noarch
openstack-ceilometer-api-2013.2.3-2.el6.noarch
openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
openstack-nova-scheduler-2013.2.3-1.el6.noarch
openstack-ceilometer-common-2013.2.3-2.el6.noarch
python-django-openstack-auth-1.1.2-1.el6.noarch
openstack-ceilometer-central-2013.2.3-2.el6.noarch
openstack-ceilometer-collector-2013.2.3-2.el6.noarch
openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
openstack-nova-common-2013.2.3-1.el6.noarch
openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
openstack-glance-2013.2.3-2.el6.noarch
openstack-nova-conductor-2013.2.3-1.el6.noarch
openstack-nova-novncproxy-2013.2.3-1.el6.noarch
openstack-nova-cert-2013.2.3-1.el6.noarch
openstack-keystone-2013.2.3-3.el6.noarch
openstack-neutron-2013.2.3-9.el6.noarch
openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
openstack-dashboard-2013.2.3-1.el6.noarch
openstack-nova-api-2013.2.3-1.el6.noarch
openstack-nova-console-2013.2.3-1.el6.noarch
openstack-utils-2013.2-2.el6.noarch

~ rpm -qa | grep -i openstack - On Compute Node
openstack-ceilometer-common-2013.2.3-2.el6.noarch
openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
openstack-utils-2013.2-2.el6.noarch
openstack-ceilometer-compute-2013.2.3-2.el6.noarch
openstack-nova-compute-2013.2.3-1.el6.noarch
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-nova-common-2013.2.3-1.el6.noarch
openstack-neutron-2013.2.3-9.el6.noarch

dhcp-agent is running on the controller node.

neutron  30198  0.4  0.1 275060 33732 ?        S    01:18   0:03 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

Help please.