SOLVED

So the problem was that the linux bridge interfaces on the host node were removed for whatever reason after the router failure, and those are not created by neutron, those are created by nova. So instead of messing with neutron and openvswitch, all I had to do was service openstack-nova-compute restart and then a final service neutron-openvswitch-agent restart and everything was fixed.

Original problem:

After a physical router failure we had to restart the network service on our neutron host machine. After doing this and even after restarting the neutron-openvswitch-agent and neutron-l3-agent we are unable to reach VMs on the host. This is not a problem for our other two compute nodes on the same neutron router.

I was able to figure out that openvswitch is not creating ports for the VMs on the neutron host, although they are present in /etc/openvswitch/conf.db . I cannot find errors in any logs.

In the neutron database, the ports on the neutron host have the status of "DOWN".

Is there anyway to get openvswitch to realize it needs to create the ports if restarting it does not work?

I'm not sure exactly what config files or log files are relevant, and I just hope my question isn't too messy. This is for the Mitaka release.

Here is the truncated ovs-vsctl show output on the neutron/compute node where I cannot reach VMs

    Bridge br-ex
    Port "em2"
        Interface "em2"
    Port "qg-15c8ef30-fa"
        Interface "qg-15c8ef30-fa"
            type: internal
    Port phy-br-ex
        Interface phy-br-ex
            type: patch
            options: {peer=int-br-ex}
    Port br-ex
        Interface br-ex
            type: internal
Bridge br-int
    fail_mode: secure
    Port int-br-ex
        Interface int-br-ex
            type: patch
            options: {peer=phy-br-ex}
    Port patch-tun
        Interface patch-tun
            type: patch
            options: {peer=patch-int}
    Port "tap303d9e37-5f"
        tag: 1
        Interface "tap303d9e37-5f"
            type: internal
    Port "qr-943c88cf-6c"
        tag: 1
        Interface "qr-943c88cf-6c"
            type: internal
    Port br-int
        Interface br-int
            type: internal
Bridge br-tun
    fail_mode: secure
    Port patch-int
        Interface patch-int
            type: patch
            options: {peer=patch-tun}
    Port "vxlan-c0a8040b"
        Interface "vxlan-c0a8040b"
            type: vxlan
            options {redacted}

And here is ovs-vsctl show on a compute node that does work.

    Bridge br-ex
    Port br-ex
        Interface br-ex
            type: internal
    Port phy-br-ex
        Interface phy-br-ex
            type: patch
            options: {peer=int-br-ex}
    Port "em1"
        Interface "em1"
Bridge br-int
    fail_mode: secure
    Port "qvoc3aced4c-a1"
        tag: 3
        Interface "qvoc3aced4c-a1"
    Port br-int
        Interface br-int
            type: internal
    Port patch-tun
        Interface patch-tun
            type: patch
            options: {peer=patch-int}
    Port int-br-ex
        Interface int-br-ex
            type: patch
            options: {peer=phy-br-ex}
    Port "qvo8323110f-28"
        tag: 3
        Interface "qvo8323110f-28"
    Port "qvo265e94f3-db"
        tag: 3
        Interface "qvo265e94f3-db"
Bridge br-tun
    fail_mode: secure
    Port patch-int
        Interface patch-int
            type: patch
            options: {peer=patch-tun}
    Port "vxlan-c0a8040b"
        Interface "vxlan-c0a8040b"
            type: vxlan
            options: {redacted}
    Port "vxlan-c0a80402"
        Interface "vxlan-c0a80402"
            type: vxlan
            options: {redacted}
    Port br-tun
        Interface br-tun
            type: internal
ovs_version: "2.5.0"

See how the working node creates the ports on br-int that are like "qvoxxxxxxx-xx"?

I also see in the broken node's openswitch-agent.log but not on the working nodes' log:

2016-11-17 09:43:37.253 32811 WARNING stevedore.named [req-562caef6-a0cf-4c0a-8443-d1e151216af6 - - - - -] Could not load neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

If I execute ovsdb-client dump on the ... (more)