Openvswitch agent not creating internal ports after restart
SOLVED
So the problem was that the linux bridge interfaces on the host node were removed for whatever reason after the router failure, and those are not created by neutron, those are created by nova. So instead of messing with neutron and openvswitch, all I had to do was service openstack-nova-compute restart
and then a final service neutron-openvswitch-agent restart
and everything was fixed.
Original problem:
After a physical router failure we had to restart the network service on our neutron host machine. After doing this and even after restarting the neutron-openvswitch-agent and neutron-l3-agent we are unable to reach VMs on the host. This is not a problem for our other two compute nodes on the same neutron router.
I was able to figure out that openvswitch is not creating ports for the VMs on the neutron host, although they are present in /etc/openvswitch/conf.db . I cannot find errors in any logs.
In the neutron database, the ports on the neutron host have the status of "DOWN".
Is there anyway to get openvswitch to realize it needs to create the ports if restarting it does not work?
I'm not sure exactly what config files or log files are relevant, and I just hope my question isn't too messy. This is for the Mitaka release.
Here is the truncated ovs-vsctl show
output on the neutron/compute node where I cannot reach VMs
Bridge br-ex
Port "em2"
Interface "em2"
Port "qg-15c8ef30-fa"
Interface "qg-15c8ef30-fa"
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port br-ex
Interface br-ex
type: internal
Bridge br-int
fail_mode: secure
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tap303d9e37-5f"
tag: 1
Interface "tap303d9e37-5f"
type: internal
Port "qr-943c88cf-6c"
tag: 1
Interface "qr-943c88cf-6c"
type: internal
Port br-int
Interface br-int
type: internal
Bridge br-tun
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-c0a8040b"
Interface "vxlan-c0a8040b"
type: vxlan
options {redacted}
And here is ovs-vsctl show
on a compute node that does work.
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port "em1"
Interface "em1"
Bridge br-int
fail_mode: secure
Port "qvoc3aced4c-a1"
tag: 3
Interface "qvoc3aced4c-a1"
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "qvo8323110f-28"
tag: 3
Interface "qvo8323110f-28"
Port "qvo265e94f3-db"
tag: 3
Interface "qvo265e94f3-db"
Bridge br-tun
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "vxlan-c0a8040b"
Interface "vxlan-c0a8040b"
type: vxlan
options: {redacted}
Port "vxlan-c0a80402"
Interface "vxlan-c0a80402"
type: vxlan
options: {redacted}
Port br-tun
Interface br-tun
type: internal
ovs_version: "2.5.0"
See how the working node creates the ports on br-int
that are like "qvoxxxxxxx-xx"?
I also see in the broken node's openswitch-agent.log but not on the working nodes' log:
2016-11-17 09:43:37.253 32811 WARNING stevedore.named [req-562caef6-a0cf-4c0a-8443-d1e151216af6 - - - - -] Could not load neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
If I execute ovsdb-client dump
on the ...
Post
ovs-vsctl show
on Network Node ( running Neutron Server ) andovs-vsctl show
on Compute Node. There is an issue with interprocess communication Neutron Server <=> OVS agentI've just edited my question to include them.
I also just noticed this is in /var/log/neutron/openvswitch-agent.log
Post openvswitch_agent.ini on both nodes and neutron.conf from Controller Node
Take a look at https://bugs.launchpad.net/neutron/+b...