Revision history [back]

click to hide/show revision 1
initial version

neutron OVS bridge interface not pinging to Gateway & instances not getting DHCP IP

Hi,

I have installed with 1 controller, compute, neutron with OVS single flat network.

Issue : When i assign ipaddress to bridge it is not pinging to the router/gateway. Also if i create instance it is not able to get DHCP ipaddress and VM not able to reach outside.

Have cross verified all the configuration all seems good.

Can you please let me know what is the reason it is failing!!!..

Compute 1 : Logs

Openvswitch-agent.log

2014-04-01 12:00:38.217 30056 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface']. Exception: Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] Exit code: 1 Stdout: '' Stderr: '2014-04-01T06:30:38Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n' 2014-04-01 12:00:38.624 30056 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-ofctl', 'del-flows', 'br-int', 'in_port=5,dl_vlan=1']. Exception: Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-ofctl', 'del-flows', 'br-int', 'in_port=5,dl_vlan=1'] Exit code: 1 Stdout: '' Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'

Ovs-ctl.log

root@compute1:/var/log/openvswitch# tail ovs-ctl.log * Enabling remote OVSDB managers Tue Apr 1 06:30:37 UTC 2014:stop * Killing ovs-vswitchd (23863) * Killing ovsdb-server (23854) Tue Apr 1 06:30:38 UTC 2014:load-kmod Tue Apr 1 06:30:38 UTC 2014:start --system-id=random * Starting ovsdb-server * Configuring Open vSwitch system IDs * Starting ovs-vswitchd * Enabling remote OVSDB managers

Ovs-vswitchd.log

root@compute1:/var/log/openvswitch# tail ovs-vswitchd.log 2014-04-01T07:03:19Z|00124|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap82f2e500-ab failed: No such device 2014-04-01T07:03:19Z|00125|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapd50d38a4-c8 failed: No such device 2014-04-01T07:03:19Z|00126|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap7b2400d0-01 failed: No such device 2014-04-01T07:03:19Z|00127|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapd8df9162-d5 failed: No such device 2014-04-01T07:03:19Z|00128|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap73f0c572-da failed: No such device 2014-04-01T07:03:19Z|00129|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap3fd13794-33 failed: No such device 2014-04-01T07:03:19Z|00130|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device int-br-eth01 failed: No such device 2014-04-01T07:03:19Z|00131|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapec6cb69f-4d failed: No such device 2014-04-01T07:03:19Z|00132|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap57c6ff29-c7 failed: No such device 2014-04-01T07:03:30Z|00133|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes)

Controller:- (10.37.39.38)

Server.log

2014-04-01 12:31:13.406 31199 WARNING neutron.openstack.common.notifier.rabbit_notifier [-] Deprecated: The rabbit_notifier is now deprecated. Please use rpc_notifier instead. 2014-04-01 12:31:13.432 31199 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'52c2e356-4140-49ec-a1e3-627a1b727b16'], 'name': u'sharednet1', 'provider:physical_network': u'physnet1', 'admin_state_up': True, 'tenant_id': u'3aa1f32f4f4748b9aacc96629b49cfe3', 'provider:network_type': u'flat', 'router:external': False, 'shared': True, 'id': u'8b9ba025-df0b-4340-992c-0c50c8e07c2e', 'provider:segmentation_id': None}

Keystone.log 2014-04-01 20:32:49.508 23923 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 10.37.39.76 2014-04-01 20:42:49.606 23923 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 10.37.39.76

Nova-cert.log

root@controller:/var/log/nova# tail nova-cert.log 2014-04-01 11:59:42.961 951 INFO nova.openstack.common.service [-] Caught SIGTERM, exiting 2014-04-01 11:59:43.984 31273 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2014-04-01 11:59:44.029 31273 AUDIT nova.service [-] Starting cert node (version 2013.2.2) 2014-04-01 11:59:44.653 31273 INFO nova.openstack.common.rpc.common [req-1c26d6f6-51d2-421c-9184-e98653c541de None None] Connected to AMQP server on controller:5672

Neutron : (10.37.39.40)

Ovs-vswitchd.log

root@neutron:/var/log/openvswitch# tail ovs-vswitchd.log 2014-04-01T06:31:01Z|05860|ofproto|INFO|br-int: 3 flow_mods 10 s ago (3 deletes) 2014-04-01T06:31:31Z|05861|bridge|INFO|bridge br-int: added interface int-br-int on port 5 2014-04-01T06:31:31Z|05862|bridge|INFO|bridge br-int: added interface phy-br-int on port 6 2014-04-01T06:32:01Z|05863|ofproto|INFO|br-int: 9 flow_mods in the 2 s starting 30 s ago (6 adds, 3 deletes) 2014-04-01T06:36:27Z|05864|ofproto|INFO|br-int: 2 flow_mods in the 7 s starting 10 s ago (2 deletes) 2014-04-01T06:37:27Z|05865|ofproto|INFO|br-int: 1 flow_mods 15 s ago (1 deletes) 2014-04-01T06:42:18Z|05866|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T06:49:08Z|05867|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T06:52:54Z|05868|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T07:01:23Z|05869|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)

Dhcp-agent.log

root@neutron:/var/log/neutron# tail dhcp-agent.log.1 2014-03-28 10:51:11.959 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 7 seconds. 2014-03-28 10:51:21.970 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 9 seconds. 2014-03-28 10:51:21.971 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 9 seconds. 2014-03-28 10:51:33.982 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 11 seconds. 2014-03-28 10:51:33.983 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 11 seconds. 2014-03-28 10:51:47.990 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 13 seconds. 2014-03-28 10:51:47.991 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 13 seconds. 2014-03-28 10:52:01.014 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds. 2014-03-28 10:52:01.015 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNRE

Openvswitch-agent.log

root@neutron:/var/log/neutron# tail openvswitch-agent.log 2014-04-01 11:58:46.494 29510 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:47.143 29531 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:47.798 29552 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:48.450 29573 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:49.107 29594 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:49.754 29615 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:50.404 29636 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:51.063 29657 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:51.716 29678 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:52.371 29699 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated!

Server.log 2014-04-01 12:01:20.765 30722 TRACE neutron.service raise RuntimeError(msg) 2014-04-01 12:01:20.765 30722 TRACE neutron.service RuntimeError: Unable to load quantum from configuration file /etc/neutron/api-paste.ini. 2014-04-01 12:01:20.765 30722 TRACE neutron.service

neutron OVS bridge interface not pinging to Gateway & instances not getting DHCP IP

Hi,

I have installed with 1 controller, compute, neutron with OVS single flat network.

Issue : When i assign ipaddress to bridge it is not pinging to the router/gateway. Also if i create instance it is not able to get DHCP ipaddress and VM not able to reach outside.

Have cross verified all the configuration all seems good.

Can you please let me know what is the reason it is failing!!!..

Compute 1 : Logs

Openvswitch-agent.log

2014-04-01 12:00:38.217 30056 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface']. Exception: Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] Exit code: 1 Stdout: '' Stderr: '2014-04-01T06:30:38Z|00001|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt failed (No such file or directory)\novs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n' 2014-04-01 12:00:38.624 30056 ERROR neutron.agent.linux.ovs_lib [-] Unable to execute ['ovs-ofctl', 'del-flows', 'br-int', 'in_port=5,dl_vlan=1']. Exception: Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-ofctl', 'del-flows', 'br-int', 'in_port=5,dl_vlan=1'] Exit code: 1 Stdout: '' Stderr: 'ovs-ofctl: br-int is not a bridge or a socket\n'

Ovs-ctl.log

root@compute1:/var/log/openvswitch# tail ovs-ctl.log * Enabling remote OVSDB managers Tue Apr 1 06:30:37 UTC 2014:stop * Killing ovs-vswitchd (23863) * Killing ovsdb-server (23854) Tue Apr 1 06:30:38 UTC 2014:load-kmod Tue Apr 1 06:30:38 UTC 2014:start --system-id=random * Starting ovsdb-server * Configuring Open vSwitch system IDs * Starting ovs-vswitchd * Enabling remote OVSDB managers

Ovs-vswitchd.log

root@compute1:/var/log/openvswitch# tail ovs-vswitchd.log 2014-04-01T07:03:19Z|00124|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap82f2e500-ab failed: No such device 2014-04-01T07:03:19Z|00125|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapd50d38a4-c8 failed: No such device 2014-04-01T07:03:19Z|00126|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap7b2400d0-01 failed: No such device 2014-04-01T07:03:19Z|00127|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapd8df9162-d5 failed: No such device 2014-04-01T07:03:19Z|00128|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap73f0c572-da failed: No such device 2014-04-01T07:03:19Z|00129|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap3fd13794-33 failed: No such device 2014-04-01T07:03:19Z|00130|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device int-br-eth01 failed: No such device 2014-04-01T07:03:19Z|00131|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tapec6cb69f-4d failed: No such device 2014-04-01T07:03:19Z|00132|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device tap57c6ff29-c7 failed: No such device 2014-04-01T07:03:30Z|00133|ofproto|INFO|br-int: 3 flow_mods 10 s ago (2 adds, 1 deletes)

Controller:- (10.37.39.38)

Server.log

2014-04-01 12:31:13.406 31199 WARNING neutron.openstack.common.notifier.rabbit_notifier [-] Deprecated: The rabbit_notifier is now deprecated. Please use rpc_notifier instead. 2014-04-01 12:31:13.432 31199 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'52c2e356-4140-49ec-a1e3-627a1b727b16'], 'name': u'sharednet1', 'provider:physical_network': u'physnet1', 'admin_state_up': True, 'tenant_id': u'3aa1f32f4f4748b9aacc96629b49cfe3', 'provider:network_type': u'flat', 'router:external': False, 'shared': True, 'id': u'8b9ba025-df0b-4340-992c-0c50c8e07c2e', 'provider:segmentation_id': None}

Keystone.log 2014-04-01 20:32:49.508 23923 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 10.37.39.76 2014-04-01 20:42:49.606 23923 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 10.37.39.76

Nova-cert.log

root@controller:/var/log/nova# tail nova-cert.log 2014-04-01 11:59:42.961 951 INFO nova.openstack.common.service [-] Caught SIGTERM, exiting 2014-04-01 11:59:43.984 31273 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative 2014-04-01 11:59:44.029 31273 AUDIT nova.service [-] Starting cert node (version 2013.2.2) 2014-04-01 11:59:44.653 31273 INFO nova.openstack.common.rpc.common [req-1c26d6f6-51d2-421c-9184-e98653c541de None None] Connected to AMQP server on controller:5672

Neutron : (10.37.39.40)

Ovs-vswitchd.log

root@neutron:/var/log/openvswitch# tail ovs-vswitchd.log 2014-04-01T06:31:01Z|05860|ofproto|INFO|br-int: 3 flow_mods 10 s ago (3 deletes) 2014-04-01T06:31:31Z|05861|bridge|INFO|bridge br-int: added interface int-br-int on port 5 2014-04-01T06:31:31Z|05862|bridge|INFO|bridge br-int: added interface phy-br-int on port 6 2014-04-01T06:32:01Z|05863|ofproto|INFO|br-int: 9 flow_mods in the 2 s starting 30 s ago (6 adds, 3 deletes) 2014-04-01T06:36:27Z|05864|ofproto|INFO|br-int: 2 flow_mods in the 7 s starting 10 s ago (2 deletes) 2014-04-01T06:37:27Z|05865|ofproto|INFO|br-int: 1 flow_mods 15 s ago (1 deletes) 2014-04-01T06:42:18Z|05866|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T06:49:08Z|05867|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T06:52:54Z|05868|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes) 2014-04-01T07:01:23Z|05869|ofproto|INFO|br-int: 1 flow_mods 10 s ago (1 deletes)

Dhcp-agent.log

root@neutron:/var/log/neutron# tail dhcp-agent.log.1 2014-03-28 10:51:11.959 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 7 seconds. 2014-03-28 10:51:21.970 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 9 seconds. 2014-03-28 10:51:21.971 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 9 seconds. 2014-03-28 10:51:33.982 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 11 seconds. 2014-03-28 10:51:33.983 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 11 seconds. 2014-03-28 10:51:47.990 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 13 seconds. 2014-03-28 10:51:47.991 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 13 seconds. 2014-03-28 10:52:01.014 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds. 2014-03-28 10:52:01.015 1185 ERROR neutron.openstack.common.rpc.common [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNRE

Openvswitch-agent.log

root@neutron:/var/log/neutron# tail openvswitch-agent.log 2014-04-01 11:58:46.494 29510 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:47.143 29531 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:47.798 29552 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:48.450 29573 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:49.107 29594 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:49.754 29615 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:50.404 29636 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:51.063 29657 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:51.716 29678 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated! 2014-04-01 11:58:52.371 29699 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Bridge br-eth01 for physical network physnet1 does not exist. Agent terminated!

Server.log 2014-04-01 12:01:20.765 30722 TRACE neutron.service raise RuntimeError(msg) 2014-04-01 12:01:20.765 30722 TRACE neutron.service RuntimeError: Unable to load quantum from configuration file /etc/neutron/api-paste.ini. 2014-04-01 12:01:20.765 30722 TRACE neutron.service