Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Instances unable to reach DHCP, ping other instances,router.

I have been stuck for about a week and have gone through every troubleshooter i could find. The instances could not reach the dhcp so they self assigned an ipv6 address. I assigned a static ipv4 that they were supposed to have and they can ping another instance as long as they are on the same node. They are unable to ping the router or anything outside of it. Using ip netns exec to get into the routers namespace I am unable to ping the machines on the network or outside the network as well. I was unable to find anything in the error logs except the neutron one mentioned below.

Controller: ovs-vsctl show

d3554148-bd29-4011-9dc2-f4bfe5b98d02 Bridge "br-eth1" Port "eth1" Interface "eth1" Port "br-eth1" Interface "br-eth1" type: internal Port "phy-br-eth1" Interface "phy-br-eth1" Bridge br-int Port "qg-89453d16-b7" tag: 4095 Interface "qg-89453d16-b7" type: internal Port "qr-b143c6db-76" tag: 4095 Interface "qr-b143c6db-76" type: internal Port "qr-26fed736-df" tag: 5 Interface "qr-26fed736-df" type: internal Port br-int Interface br-int type: internal Port "tap77edd713-fb" tag: 5 Interface "tap77edd713-fb" type: internal Port "int-br-eth1" Interface "int-br-eth1" Port "qg-4903a187-55" tag: 2 Interface "qg-4903a187-55" type: internal Port "tap9329b3bb-92" tag: 2 Interface "tap9329b3bb-92" type: internal ovs_version: "1.10.2"

ip a | grep state

1: lo: <loopback,up,lower_up> mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 3: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 16: vip-br: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 18: virbr0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state DOWN 20: phy-br-eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 21: int-br-eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 31: ovs-system: <broadcast,multicast> mtu 1500 qdisc noop state DOWN 32: br-eth1: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 33: tap9329b3bb-92: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 34: br-int: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 35: qr-b143c6db-76: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 36: qr-26fed736-df: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 37: qg-4903a187-55: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 38: tap77edd713-fb: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 39: qg-89453d16-b7: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN

Compute node: ovs-vsctl show

a85aec19-2017-439a-86ba-e601a43b442e Bridge br-ex Port br-ex Interface br-ex type: internal Bridge br-int Port "qvoe0600e53-f5" tag: 3 Interface "qvoe0600e53-f5" Port "int-br-eth1" Interface "int-br-eth1" Port "qvof3293c68-b9" tag: 3 Interface "qvof3293c68-b9" Port br-int Interface br-int type: internal Bridge "br-eth1" Port "br-eth1" Interface "br-eth1" type: internal Port "phy-br-eth1" Interface "phy-br-eth1" Port "eth1" Interface "eth1" ovs_version: "1.10.2"

ip a | grep state

1: lo: <loopback,up,lower_up> mtu 16436 qdisc noqueue state UNKNOWN 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state UP qlen 1000 3: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state UP qlen 1000 4: ovs-system: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 5: br-eth1: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 6: br-ex: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 7: br-int: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 9: phy-br-eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 10: int-br-eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 22: qbre0600e53-f5: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP 23: qvoe0600e53-f5: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 24: qvbe0600e53-f5: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UP qlen 1000 25: tape0600e53-f5: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UNKNOWN qlen 500 26: qbrf3293c68-b9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state UP 27: qvof3293c68-b9: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000 28: qvbf3293c68-b9: <broadcast,multicast,promisc,up,lower_up> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UP qlen 1000 29: tapf3293c68-b9: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UNKNOWN qlen 500

All logs are good except the neutron/openvswitch-agent.log that has this every couple seconds:

neutron/openvswitch.log

2014-06-17 20:30:53.583 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47 2014-06-17 20:30:53.681 1026 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int'] Exit code: 0 Stdout: 'int-br-eth1\nqg-4903a187-55\nqg-89453d16-b7\nqr-26fed736-df\nqr-b143c6db-76\ntap77edd713-fb\ntap9329b3bb-92\n' Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73 2014-06-17 20:30:53.683 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47 2014-06-17 20:30:53.780 1026 DEBUG neutron.agent.linux.utils [-] Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] Exit code: 0 Stdout: '{"data":[["qr-b143c6db-76",["map",[["attached-mac","fa:16:3e:b9:e9:98"],["iface-id","b143c6db-7625-4ac1-b29e-a1f158b4d971"],["iface-status","active"]]]],["br-eth1",["map",[]]],["tap9329b3bb-92",["map",[["attached-mac","fa:16:3e:ff:fc:7b"],["iface-id","9329b3bb-92db-48d5-8a01-c3ac8557fe4a"],["iface-status","active"]]]],["int-br-eth1",["map",[]]],["qg-4903a187-55",["map",[["attached-mac","fa:16:3e:08:0d:b7"],["iface-id","4903a187-55ee-46f1-be69-88579cf0ff8e"],["iface-status","active"]]]],["qg-89453d16-b7",["map",[["attached-mac","fa:16:3e:67:30:38"],["iface-id","89453d16-b796-4ed4-8c12-0456b419d141"],["iface-status","active"]]]],["phy-br-eth1",["map",[]]],["qr-26fed736-df",["map",[["attached-mac","fa:16:3e:c3:27:cc"],["iface-id","26fed736-df83-4a68-b0ce-c4efad56f979"],["iface-status","active"]]]],["tap77edd713-fb",["map",[["attached-mac","fa:16:3e:de:a9:db"],["iface-id","77edd713-fbf8-43db-9149-7060d547ea6c"],["iface-status","active"]]]],["eth1",["map",[]]],["br-int",["map",[]]]],"headings":["name","external_ids"]}\n' Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73 2014-06-17 20:30:53.890 1026 DEBUG neutron.openstack.common.rpc.amqp [-] Making asynchronous cast on q-plugin... cast /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:559 2014-06-17 20:30:53.891 1026 DEBUG neutron.openstack.common.rpc.amqp [-] UNIQUE_ID is 8a58632ca49946ceb4501fd4ee90c5d4. _add_unique_id /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:339 2014-06-17 20:30:53.900 1026 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88 2014-06-17 20:30:53.900 1026 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70 2014-06-17 20:30:53.902 1026 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420

Instances unable to reach DHCP, ping other instances,router.

I have been stuck for about a week and have gone through every troubleshooter i could find. The instances could not reach the dhcp so they self assigned an ipv6 address. I assigned a static ipv4 that they were supposed to have and they can ping another instance as long as they are on the same node. They are unable to ping the router or anything outside of it. Using ip netns exec to get into the routers namespace I am unable to ping the machines on the network or outside the network as well. I was unable to find anything in the error logs except the neutron one mentioned below.

Controller:
#Controller: ovs-vsctl show

show d3554148-bd29-4011-9dc2-f4bfe5b98d02 Bridge "br-eth1" Port "eth1" Interface "eth1" Port "br-eth1" Interface "br-eth1" type: internal Port "phy-br-eth1" Interface "phy-br-eth1" Bridge br-int Port "qg-89453d16-b7" tag: 4095 Interface "qg-89453d16-b7" type: internal Port "qr-b143c6db-76" tag: 4095 Interface "qr-b143c6db-76" type: internal Port "qr-26fed736-df" tag: 5 Interface "qr-26fed736-df" type: internal Port br-int Interface br-int type: internal Port "tap77edd713-fb" tag: 5 Interface "tap77edd713-fb" type: internal Port "int-br-eth1" Interface "int-br-eth1" Port "qg-4903a187-55" tag: 2 Interface "qg-4903a187-55" type: internal Port "tap9329b3bb-92" tag: 2 Interface "tap9329b3bb-92" type: internal ovs_version: "1.10.2"

"1.10.2"

ip a | grep state

1: lo: <loopback,up,lower_up> <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
2: eth0: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
3: eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
16: vip-br: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
18: virbr0: <no-carrier,broadcast,multicast,up> <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
20: phy-br-eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
21: int-br-eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
31: ovs-system: <broadcast,multicast> <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
32: br-eth1: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
33: tap9329b3bb-92: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
34: br-int: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
35: qr-b143c6db-76: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
36: qr-26fed736-df: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
37: qg-4903a187-55: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
38: tap77edd713-fb: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
39: qg-89453d16-b7: <broadcast,up,lower_up> mtu 1500 qdisc noqueue state UNKNOWN 

<BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

Compute node: ovs-vsctl show

a85aec19-2017-439a-86ba-e601a43b442e
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        Port "qvoe0600e53-f5"
            tag: 3
            Interface "qvoe0600e53-f5"
        Port "int-br-eth1"
            Interface "int-br-eth1"
        Port "qvof3293c68-b9"
            tag: 3
            Interface "qvof3293c68-b9"
        Port br-int
            Interface br-int
                type: internal
    Bridge "br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "phy-br-eth1"
            Interface "phy-br-eth1"
        Port "eth1"
            Interface "eth1"
    ovs_version: "1.10.2"

"1.10.2"

ip a | grep state

1: lo: <loopback,up,lower_up> <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
2: eth0: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
3: eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
4: ovs-system: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
5: br-eth1: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
6: br-ex: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
7: br-int: <broadcast,up,lower_up> <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
9: phy-br-eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
10: int-br-eth1: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
22: qbre0600e53-f5: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
23: qvoe0600e53-f5: <broadcast,multicast,promisc,up,lower_up> <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
24: qvbe0600e53-f5: <broadcast,multicast,promisc,up,lower_up> <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UP qlen 1000
25: tape0600e53-f5: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UNKNOWN qlen 500
26: qbrf3293c68-b9: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
27: qvof3293c68-b9: <broadcast,multicast,promisc,up,lower_up> <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
28: qvbf3293c68-b9: <broadcast,multicast,promisc,up,lower_up> <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UP qlen 1000
29: tapf3293c68-b9: <broadcast,multicast,up,lower_up> <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UNKNOWN qlen 500

500

All logs are good except the neutron/openvswitch-agent.log that has this every couple seconds:

syslog

Jun 18 17:30:37 os-controller-1.xxx.xxx.xxx monit[3178]: 'neutron-lbaas-agent' failed to start

neutron/openvswitch.log

2014-06-17 20:30:53.583 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47
2014-06-17 20:30:53.681 1026 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
Exit code: 0
Stdout: 'int-br-eth1\nqg-4903a187-55\nqg-89453d16-b7\nqr-26fed736-df\nqr-b143c6db-76\ntap77edd713-fb\ntap9329b3bb-92\n'
Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73
2014-06-17 20:30:53.683 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47
2014-06-17 20:30:53.780 1026 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface']
Exit code: 0
Stdout: '{"data":[["qr-b143c6db-76",["map",[["attached-mac","fa:16:3e:b9:e9:98"],["iface-id","b143c6db-7625-4ac1-b29e-a1f158b4d971"],["iface-status","active"]]]],["br-eth1",["map",[]]],["tap9329b3bb-92",["map",[["attached-mac","fa:16:3e:ff:fc:7b"],["iface-id","9329b3bb-92db-48d5-8a01-c3ac8557fe4a"],["iface-status","active"]]]],["int-br-eth1",["map",[]]],["qg-4903a187-55",["map",[["attached-mac","fa:16:3e:08:0d:b7"],["iface-id","4903a187-55ee-46f1-be69-88579cf0ff8e"],["iface-status","active"]]]],["qg-89453d16-b7",["map",[["attached-mac","fa:16:3e:67:30:38"],["iface-id","89453d16-b796-4ed4-8c12-0456b419d141"],["iface-status","active"]]]],["phy-br-eth1",["map",[]]],["qr-26fed736-df",["map",[["attached-mac","fa:16:3e:c3:27:cc"],["iface-id","26fed736-df83-4a68-b0ce-c4efad56f979"],["iface-status","active"]]]],["tap77edd713-fb",["map",[["attached-mac","fa:16:3e:de:a9:db"],["iface-id","77edd713-fbf8-43db-9149-7060d547ea6c"],["iface-status","active"]]]],["eth1",["map",[]]],["br-int",["map",[]]]],"headings":["name","external_ids"]}\n'
Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73
2014-06-17 20:30:53.890 1026 DEBUG neutron.openstack.common.rpc.amqp [-] Making asynchronous cast on q-plugin... cast /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:559
2014-06-17 20:30:53.891 1026 DEBUG neutron.openstack.common.rpc.amqp [-] UNIQUE_ID is 8a58632ca49946ceb4501fd4ee90c5d4. _add_unique_id /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:339
2014-06-17 20:30:53.900 1026 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
2014-06-17 20:30:53.900 1026 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
2014-06-17 20:30:53.902 1026 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420

/usr/lib/python2.7/dist-packages/amqp/channel.py:420

Instances unable to reach DHCP, ping other instances,router.

I have been stuck for about a week and have gone through every troubleshooter i could find. The instances could not reach the dhcp so they self assigned an ipv6 address. I assigned a static ipv4 that they were supposed to have and they can ping another instance as long as they are on the same node. They are unable to ping the router or anything outside of it. Using ip netns exec to get into the routers namespace I am unable to ping the machines on the network or outside the network as well. I was unable to find anything in the error logs except the neutron one mentioned below.

Controller: ovs-vsctl show

#Controller: ovs-vsctl show
 d3554148-bd29-4011-9dc2-f4bfe5b98d02
     Bridge "br-eth1"
         Port "eth1"
             Interface "eth1"
         Port "br-eth1"
             Interface "br-eth1"
                 type: internal
         Port "phy-br-eth1"
             Interface "phy-br-eth1"
     Bridge br-int
         Port "qg-89453d16-b7"
             tag: 4095
             Interface "qg-89453d16-b7"
                 type: internal
         Port "qr-b143c6db-76"
             tag: 4095
             Interface "qr-b143c6db-76"
                 type: internal
         Port "qr-26fed736-df"
             tag: 5
             Interface "qr-26fed736-df"
                 type: internal
         Port br-int
             Interface br-int
                 type: internal
         Port "tap77edd713-fb"
             tag: 5
             Interface "tap77edd713-fb"
                 type: internal
         Port "int-br-eth1"
             Interface "int-br-eth1"
         Port "qg-4903a187-55"
             tag: 2
             Interface "qg-4903a187-55"
                 type: internal
         Port "tap9329b3bb-92"
             tag: 2
             Interface "tap9329b3bb-92"
                 type: internal
     ovs_version: "1.10.2"

ip a | grep state

 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 16: vip-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 18: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
 20: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 21: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 31: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
 32: br-eth1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 33: tap9329b3bb-92: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 34: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 35: qr-b143c6db-76: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 36: qr-26fed736-df: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 37: qg-4903a187-55: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 38: tap77edd713-fb: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
 39: qg-89453d16-b7: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN

Compute node: ovs-vsctl show

 a85aec19-2017-439a-86ba-e601a43b442e
     Bridge br-ex
         Port br-ex
             Interface br-ex
                 type: internal
     Bridge br-int
         Port "qvoe0600e53-f5"
             tag: 3
             Interface "qvoe0600e53-f5"
         Port "int-br-eth1"
             Interface "int-br-eth1"
         Port "qvof3293c68-b9"
             tag: 3
             Interface "qvof3293c68-b9"
         Port br-int
             Interface br-int
                 type: internal
     Bridge "br-eth1"
         Port "br-eth1"
             Interface "br-eth1"
                 type: internal
         Port "phy-br-eth1"
             Interface "phy-br-eth1"
         Port "eth1"
             Interface "eth1"
     ovs_version: "1.10.2"

ip a | grep state

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
4: ovs-system: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
5: br-eth1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
6: br-ex: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
7: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
9: phy-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
10: int-br-eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
22: qbre0600e53-f5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
23: qvoe0600e53-f5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
24: qvbe0600e53-f5: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UP qlen 1000
25: tape0600e53-f5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbre0600e53-f5 state UNKNOWN qlen 500
26: qbrf3293c68-b9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
27: qvof3293c68-b9: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
28: qvbf3293c68-b9: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UP qlen 1000
29: tapf3293c68-b9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbrf3293c68-b9 state UNKNOWN qlen 500

All logs are good except the neutron/openvswitch-agent.log that has this every couple seconds:

syslog

Jun 18 17:30:37 os-controller-1.xxx.xxx.xxx monit[3178]: 'neutron-lbaas-agent' failed to start

neutron/openvswitch.log

var/log/neutron/openvswitch.log

2014-06-17 20:30:53.583 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47
2014-06-17 20:30:53.681 1026 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
Exit code: 0
Stdout: 'int-br-eth1\nqg-4903a187-55\nqg-89453d16-b7\nqr-26fed736-df\nqr-b143c6db-76\ntap77edd713-fb\ntap9329b3bb-92\n'
Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73
2014-06-17 20:30:53.683 1026 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:47
2014-06-17 20:30:53.780 1026 DEBUG neutron.agent.linux.utils [-] 
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=2', '--format=json', '--', '--columns=name,external_ids', 'list', 'Interface']
Exit code: 0
Stdout: '{"data":[["qr-b143c6db-76",["map",[["attached-mac","fa:16:3e:b9:e9:98"],["iface-id","b143c6db-7625-4ac1-b29e-a1f158b4d971"],["iface-status","active"]]]],["br-eth1",["map",[]]],["tap9329b3bb-92",["map",[["attached-mac","fa:16:3e:ff:fc:7b"],["iface-id","9329b3bb-92db-48d5-8a01-c3ac8557fe4a"],["iface-status","active"]]]],["int-br-eth1",["map",[]]],["qg-4903a187-55",["map",[["attached-mac","fa:16:3e:08:0d:b7"],["iface-id","4903a187-55ee-46f1-be69-88579cf0ff8e"],["iface-status","active"]]]],["qg-89453d16-b7",["map",[["attached-mac","fa:16:3e:67:30:38"],["iface-id","89453d16-b796-4ed4-8c12-0456b419d141"],["iface-status","active"]]]],["phy-br-eth1",["map",[]]],["qr-26fed736-df",["map",[["attached-mac","fa:16:3e:c3:27:cc"],["iface-id","26fed736-df83-4a68-b0ce-c4efad56f979"],["iface-status","active"]]]],["tap77edd713-fb",["map",[["attached-mac","fa:16:3e:de:a9:db"],["iface-id","77edd713-fbf8-43db-9149-7060d547ea6c"],["iface-status","active"]]]],["eth1",["map",[]]],["br-int",["map",[]]]],"headings":["name","external_ids"]}\n'
Stderr: '' execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:73
2014-06-17 20:30:53.890 1026 DEBUG neutron.openstack.common.rpc.amqp [-] Making asynchronous cast on q-plugin... cast /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:559
2014-06-17 20:30:53.891 1026 DEBUG neutron.openstack.common.rpc.amqp [-] UNIQUE_ID is 8a58632ca49946ceb4501fd4ee90c5d4. _add_unique_id /usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/amqp.py:339
2014-06-17 20:30:53.900 1026 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
2014-06-17 20:30:53.900 1026 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
2014-06-17 20:30:53.902 1026 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420