Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

VMs on compute node unable to use network

Hi, I am having an issue with neutron on my compute nodes. I am using mitaka on SL7 with linux bridge networking.

I had previously been having this problem https://ask.openstack.org/en/question/94395/linux-bridge-rtnetlink-exists/ which I have resolved. Now my VMs on my flat public network are unable to get dhcp or if set with a static ip cannot ping anything. The same is true for VMs on a private network.

The DHCP agent and other agents on the network node are reachable from the compute node and other locations on the network.

I have statically assigned an ip address to a VM on the compute node and am unable to ping the compute node, the gateway or anything else on the network.

Below are config files: linuxbridge_agent.ini

[agent]
prevent_arp_spoofing=false

[linux_bridge]
bridge_mappings=public:br0
physical_interface_mappings=public:br0

[securitygroup]
enable_security_group=True
firewall_driver=neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

[vxlan]
enable_vxlan=True
l2_population=True
local_ip=130.246.223.142

output of brctl show

bridge name     bridge id               STP enabled     interfaces
br0             8000.a0369f32db38       no              p1p1
brq5a97f9b0-0f          8000.fecaad283f39       no              tap46a6bf63-f8
                                                        tap4f8b5d7a-9d

output of ifconfig

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 130.246.223.142  netmask 255.255.248.0  broadcast 130.246.223.255
        inet6 fe80::a236:9fff:fe32:db38  prefixlen 64  scopeid 0x20<link>
        ether a0:36:9f:32:db:38  txqueuelen 0  (Ethernet)
        RX packets 3026204  bytes 1402700536 (1.3 GiB)
        RX errors 0  dropped 4274  overruns 0  frame 0
        TX packets 248857  bytes 876764427 (836.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

brq5a97f9b0-0f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::6cdc:78ff:fee7:ccbb  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:28:3f:39  txqueuelen 0  (Ethernet)
        RX packets 261  bytes 11132 (10.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1376 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 137153  bytes 7279941 (6.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 137153  bytes 7279941 (6.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p1p1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether a0:36:9f:32:db:38  txqueuelen 1000  (Ethernet)
        RX packets 4106724  bytes 1621228855 (1.5 GiB)
        RX errors 0  dropped 291  overruns 0  frame 0
        TX packets 807762  bytes 913750540 (871.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap46a6bf63-f8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcca:adff:fe28:3f39  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:28:3f:39  txqueuelen 500  (Ethernet)
        RX packets 9  bytes 1458 (1.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 439  bytes 60406 (58.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap4f8b5d7a-9d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcca:adff:fe52:9754  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:52:97:54  txqueuelen 500  (Ethernet)
        RX packets 240  bytes 12512 (12.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 246  bytes 51700 (50.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

output of tcpdump -i br0 icmp on compute node while attempting to ping VM from compute node:

13:37:00.854724 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 1, length 64
13:37:01.854684 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 2, length 64
13:37:02.854704 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 3, length 64
13:37:03.854698 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 4, length 64
13:37:04.854732 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 5, length 64

VMs on compute node unable to use network

Hi, I am having an issue with neutron on my compute nodes. I am using mitaka on SL7 with linux bridge networking.

I had previously been having this problem https://ask.openstack.org/en/question/94395/linux-bridge-rtnetlink-exists/ which I have resolved. Now my VMs on my flat public network are unable to get dhcp or if set with a static ip cannot ping anything. The same is true for VMs on a private network.

The DHCP agent and other agents on the network node are reachable from the compute node and other locations on the network.

I have statically assigned an ip address to a VM on the compute node and am unable to ping the compute node, the gateway or anything else on the network.

Below are config files: linuxbridge_agent.ini

[agent]
prevent_arp_spoofing=false

[linux_bridge]
bridge_mappings=public:br0
physical_interface_mappings=public:br0

[securitygroup]
enable_security_group=True
firewall_driver=neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

[vxlan]
enable_vxlan=True
l2_population=True
local_ip=130.246.223.142

output of brctl show

bridge name     bridge id               STP enabled     interfaces
br0             8000.a0369f32db38       no              p1p1
brq5a97f9b0-0f          8000.fecaad283f39       no              tap46a6bf63-f8
                                                        tap4f8b5d7a-9d

output of ifconfig

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 130.246.223.142  netmask 255.255.248.0  broadcast 130.246.223.255
        inet6 fe80::a236:9fff:fe32:db38  prefixlen 64  scopeid 0x20<link>
        ether a0:36:9f:32:db:38  txqueuelen 0  (Ethernet)
        RX packets 3026204  bytes 1402700536 (1.3 GiB)
        RX errors 0  dropped 4274  overruns 0  frame 0
        TX packets 248857  bytes 876764427 (836.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

brq5a97f9b0-0f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::6cdc:78ff:fee7:ccbb  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:28:3f:39  txqueuelen 0  (Ethernet)
        RX packets 261  bytes 11132 (10.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1376 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 137153  bytes 7279941 (6.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 137153  bytes 7279941 (6.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p1p1: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether a0:36:9f:32:db:38  txqueuelen 1000  (Ethernet)
        RX packets 4106724  bytes 1621228855 (1.5 GiB)
        RX errors 0  dropped 291  overruns 0  frame 0
        TX packets 807762  bytes 913750540 (871.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap46a6bf63-f8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcca:adff:fe28:3f39  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:28:3f:39  txqueuelen 500  (Ethernet)
        RX packets 9  bytes 1458 (1.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 439  bytes 60406 (58.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap4f8b5d7a-9d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fcca:adff:fe52:9754  prefixlen 64  scopeid 0x20<link>
        ether fe:ca:ad:52:97:54  txqueuelen 500  (Ethernet)
        RX packets 240  bytes 12512 (12.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 246  bytes 51700 (50.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

output of tcpdump -i br0 icmp on compute node while attempting to ping VM from compute node:

13:37:00.854724 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 1, length 64
13:37:01.854684 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 2, length 64
13:37:02.854704 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 3, length 64
13:37:03.854698 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 4, length 64
13:37:04.854732 IP hv15.nubes.rl.ac.uk > vm546.nubes.stfc.ac.uk: ICMP echo request, id 26973, seq 5, length 64

output of iptables -L -vn

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  46M   17G neutron-linuxbri-INPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0  
  45M   16G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
 2314  195K ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    1    60 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
   11   660 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpts:5900:5999
  884 53016 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
 810K  399M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 6080

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-filter-top  all  --  *      *       0.0.0.0/0            0.0.0.0/0      
    0     0 neutron-linuxbri-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 46M packets, 10G bytes)
 pkts bytes target     prot opt in     out     source               destination
  46M   10G neutron-filter-top  all  --  *      *       0.0.0.0/0            0.0.0.0/0      
  46M   10G neutron-linuxbri-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0 

Chain neutron-filter-top (2 references)
 pkts bytes target     prot opt in     out     source               destination
  46M   10G neutron-linuxbri-local  all  --  *      *       0.0.0.0/0            0.0.0.0/0  

Chain neutron-linuxbri-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-linuxbri-scope  all  --  *      *       0.0.0.0/0            0.0.0.0/0  
    0     0 neutron-linuxbri-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap0220d48e-6b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
    0     0 neutron-linuxbri-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap0220d48e-6b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
    0     0 neutron-linuxbri-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap7480672e-76 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
    0     0 neutron-linuxbri-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap7480672e-76 --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */

Chain neutron-linuxbri-INPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-linuxbri-o0220d48e-6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap0220d48e-6b --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */
    0     0 neutron-linuxbri-o7480672e-7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap7480672e-76 --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */

Chain neutron-linuxbri-OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain neutron-linuxbri-i0220d48e-6 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
    0     0 RETURN     udp  --  *      *       192.168.117.1        0.0.0.0/0            udp spt:67 udp dpt:68
    0     0 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set NIPv4a64e8c5d-b99b-4f2b-96ea- src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
    0     0 neutron-linuxbri-sg-fallback  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-linuxbri-i7480672e-7 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
    0     0 RETURN     udp  --  *      *       130.246.186.15       0.0.0.0/0            udp spt:67 udp dpt:68
    0     0 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set NIPv4a64e8c5d-b99b-4f2b-96ea- src
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
    0     0 neutron-linuxbri-sg-fallback  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-linuxbri-local (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain neutron-linuxbri-o0220d48e-6 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     udp  --  *      *       0.0.0.0              255.255.255.255      udp spt:68 dpt:67 /* Allow DHCP client traffic. */
    0     0 neutron-linuxbri-s0220d48e-6  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:68 dpt:67 /* Allow DHCP client traffic. */
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 udp dpt:68 /* Prevent DHCP Spoofing by VM. */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
    0     0 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
    0     0 neutron-linuxbri-sg-fallback  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-linuxbri-o7480672e-7 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     udp  --  *      *       0.0.0.0              255.255.255.255      udp spt:68 dpt:67 /* Allow DHCP client traffic. */
    0     0 neutron-linuxbri-s7480672e-7  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:68 dpt:67 /* Allow DHCP client traffic. */
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 udp dpt:68 /* Prevent DHCP Spoofing by VM. */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
    0     0 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
    0     0 neutron-linuxbri-sg-fallback  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Send unmatched traffic to the fallback chain. */

Chain neutron-linuxbri-s0220d48e-6 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       192.168.117.2        0.0.0.0/0            MAC FA:CA:AD:DA:25:E9 /* Allow traffic from defined IP/MAC pairs. */
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Drop traffic without an IP/MAC allow rule. */

Chain neutron-linuxbri-s7480672e-7 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       130.246.186.20       0.0.0.0/0            MAC FA:CA:AD:08:FB:7D /* Allow traffic from defined IP/MAC pairs. */
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Drop traffic without an IP/MAC allow rule. */

Chain neutron-linuxbri-scope (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain neutron-linuxbri-sg-chain (4 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 neutron-linuxbri-i0220d48e-6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap0220d48e-6b --physdev-is-bridged /* Jump to the VM specific chain. */
    0     0 neutron-linuxbri-o0220d48e-6  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap0220d48e-6b --physdev-is-bridged /* Jump to the VM specific chain. */
    0     0 neutron-linuxbri-i7480672e-7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tap7480672e-76 --physdev-is-bridged /* Jump to the VM specific chain. */
    0     0 neutron-linuxbri-o7480672e-7  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap7480672e-76 --physdev-is-bridged /* Jump to the VM specific chain. */
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain neutron-linuxbri-sg-fallback (4 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Default drop rule for unmatched traffic. */