Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

VM traffic not properly handled on Network node

Hi Everyone, I’ve run into a common situation that my VM instances don’t get an IP address assigned. I use

Ubuntu 14.04.1 OpenStack Juno OpenVSwitch GRE tunneling

I’ve followed the getting started instructions at http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-network-node.html.

I can set that:

  1. Traffic (DHCP request) is encapsulated into GRE on the compute node (verified by TCPdump)
  2. GRE traffic is received on the network node (verified by TCPdump)
  3. DHCP traffic isn’t visible in br-tun or br-int (verified by creating two snooper interfaces as described at http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html and attaching them to br-tun and br-int respectively)

The configuration looks right:

Proper configuration in /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0

vSwitch configuration:

root@network:~# ovs-vsctl show
db4118a3-d0e8-47ee-8d95-ef174c6b95d5
    Bridge br-ex
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth2"
            Interface "eth2"
        Port "qg-c38a9aa2-73"
            Interface "qg-c38a9aa2-73"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "snooper1"
            Interface "snooper1"
        Port "gre-0a0000c1"
            Interface "gre-0a0000c1"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="10.0.0.192", out_key=flow, remote_ip="10.0.0.193"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "snooper0"
            Interface "snooper0"
        Port "tap752ac585-ff"
            tag: 4095
            Interface "tap752ac585-ff"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-495aff55-b0"
            tag: 4095
            Interface "qr-495aff55-b0"
                type: internal
    ovs_version: "2.0.2"

Segmentation ID is 1:

root@controller:~# neutron net-show --fields provider:segmentation_id demo-net
+--------------------------+-------+
| Field                    | Value |
+--------------------------+-------+
| provider:segmentation_id | 1     |
+--------------------------+-------+

However, I don’t see flows matching ID 0x1:

root@network:~# ovs-ofctl dump-flows br-tun|grep 0x1

root@network:~# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=2763.059s, table=0, n_packets=11, n_bytes=834, idle_age=2758, priority=1,in_port=1 actions=resubmit(,2)
 cookie=0x0, duration=2759.565s, table=0, n_packets=26, n_bytes=4060, idle_age=15, priority=1,in_port=2 actions=resubmit(,3)
 cookie=0x0, duration=2762.842s, table=0, n_packets=6, n_bytes=480, idle_age=2755, priority=0 actions=drop
 cookie=0x0, duration=2762.628s, table=2, n_packets=0, n_bytes=0, idle_age=2762, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=2762.414s, table=2, n_packets=11, n_bytes=834, idle_age=2758, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=2762.193s, table=3, n_packets=26, n_bytes=4060, idle_age=15, priority=0 actions=drop
 cookie=0x0, duration=2761.975s, table=4, n_packets=0, n_bytes=0, idle_age=2761, priority=0 actions=drop
 cookie=0x0, duration=2761.756s, table=10, n_packets=0, n_bytes=0, idle_age=2761, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x0, duration=2761.55s, table=20, n_packets=0, n_bytes=0, idle_age=2761, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=2761.297s, table=22, n_packets=7, n_bytes=554, idle_age=2758, priority=0 actions=drop

I’ve the two required namespaces configured:

root@network:~# ip netns
qdhcp-b43d1b41-7123-4ffa-a1aa-d25853383bdb
qrouter-f3ba7133-39ea-4492-a746-d2e78c31593a

Dnsmasq binding seems right to me:

root@network:~# ps aux | grep dnsmasq
nobody    1714  0.0  0.1  28200   996 ?        S    13:26   0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap752ac585-ff --except-interface=lo --pid-file=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/host --addn-hosts=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/opts --leasefile-ro --dhcp-authoritative --dhcp-range=set:tag0,192.168.1.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
root      6693  0.0  0.1  11740   932 pts/7    S+   14:14   0:00 grep --color=auto dnsmasq

Agent list seems also fine to me:

root@controller:~/download# neutron agent-list
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| id                                   | agent_type         | host     | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| 61db8227-c0d1-4802-950a-ae33e8f211a8 | Open vSwitch agent | network  | :-)   | True           | neutron-openvswitch-agent |
| 8a5d36f0-a72f-4f2a-917d-082a25c620bf | L3 agent           | network  | :-)   | True           | neutron-l3-agent          |
| a42998a5-12d3-4218-a2e2-36d1d59c49e8 | Metadata agent     | network  | :-)   | True           | neutron-metadata-agent    |
| c6658c6b-7283-4984-922b-08898741889c | DHCP agent         | network  | :-)   | True           | neutron-dhcp-agent        |
| ec2cd87b-bb19-42ea-874a-f885d3d2152f | Open vSwitch agent | compute1 | :-)   | True           | neutron-openvswitch-agent |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+

Any ideas?

Best regards, Dominik

VM traffic not properly handled on Network node

Hi Everyone, I’ve run into a common situation that my VM instances don’t get an IP address assigned. I use

Ubuntu 14.04.1 OpenStack Juno OpenVSwitch GRE tunneling

I’ve followed the getting started instructions at http://docs.openstack.org/juno/install-guide/install/apt/content/neutron-network-node.html.

I can set that:

  1. Traffic (DHCP request) is encapsulated into GRE on the compute node (verified by TCPdump)
  2. GRE traffic is received on the network node (verified by TCPdump)
  3. DHCP traffic isn’t visible in br-tun or br-int (verified by creating two snooper interfaces as described at http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html and attaching them to br-tun and br-int respectively)

The configuration looks right:

Proper configuration in /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0

vSwitch configuration:

root@network:~# ovs-vsctl show
db4118a3-d0e8-47ee-8d95-ef174c6b95d5
    Bridge br-ex
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth2"
            Interface "eth2"
        Port "qg-c38a9aa2-73"
            Interface "qg-c38a9aa2-73"
                type: internal
    Bridge br-tun
        fail_mode: secure
        Port "snooper1"
            Interface "snooper1"
        Port "gre-0a0000c1"
            Interface "gre-0a0000c1"
                type: gre
                options: {df_default="true", in_key=flow, local_ip="10.0.0.192", out_key=flow, remote_ip="10.0.0.193"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "snooper0"
            Interface "snooper0"
        Port "tap752ac585-ff"
            tag: 4095
            Interface "tap752ac585-ff"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-495aff55-b0"
            tag: 4095
            Interface "qr-495aff55-b0"
                type: internal
    ovs_version: "2.0.2"

Segmentation ID is 1:

root@controller:~# neutron net-show --fields provider:segmentation_id demo-net
+--------------------------+-------+
| Field                    | Value |
+--------------------------+-------+
| provider:segmentation_id | 1     |
+--------------------------+-------+

However, I don’t see flows matching ID 0x1:

root@network:~# ovs-ofctl dump-flows br-tun|grep 0x1

root@network:~# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=2763.059s, table=0, n_packets=11, n_bytes=834, idle_age=2758, priority=1,in_port=1 actions=resubmit(,2)
 cookie=0x0, duration=2759.565s, table=0, n_packets=26, n_bytes=4060, idle_age=15, priority=1,in_port=2 actions=resubmit(,3)
 cookie=0x0, duration=2762.842s, table=0, n_packets=6, n_bytes=480, idle_age=2755, priority=0 actions=drop
 cookie=0x0, duration=2762.628s, table=2, n_packets=0, n_bytes=0, idle_age=2762, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x0, duration=2762.414s, table=2, n_packets=11, n_bytes=834, idle_age=2758, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x0, duration=2762.193s, table=3, n_packets=26, n_bytes=4060, idle_age=15, priority=0 actions=drop
 cookie=0x0, duration=2761.975s, table=4, n_packets=0, n_bytes=0, idle_age=2761, priority=0 actions=drop
 cookie=0x0, duration=2761.756s, table=10, n_packets=0, n_bytes=0, idle_age=2761, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x0, duration=2761.55s, table=20, n_packets=0, n_bytes=0, idle_age=2761, priority=0 actions=resubmit(,22)
 cookie=0x0, duration=2761.297s, table=22, n_packets=7, n_bytes=554, idle_age=2758, priority=0 actions=drop

These are the flows for br-int:

root@network:~# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=7703.129s, table=0, n_packets=0, n_bytes=0, idle_age=7703, priority=2,in_port=1 actions=drop
 cookie=0x0, duration=7712.661s, table=0, n_packets=53410, n_bytes=5173008, idle_age=1, priority=2,in_port=2 actions=drop
 cookie=0x0, duration=7703.787s, table=0, n_packets=0, n_bytes=0, idle_age=7703, priority=2,in_port=4 actions=drop
 cookie=0x0, duration=7717.155s, table=0, n_packets=94, n_bytes=6950, idle_age=7704, priority=1 actions=NORMAL
 cookie=0x0, duration=7716.655s, table=23, n_packets=0, n_bytes=0, idle_age=7716, priority=0 actions=drop

And here the corresponding interfaces:

root@network:~# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000ce41c7fbb24f
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(tap752ac585-ff): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 2(int-br-ex): addr:36:20:9f:e8:1b:09
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(patch-tun): addr:3a:6a:94:2c:92:00
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(qr-495aff55-b0): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 5(snooper0): addr:c6:0c:d3:8c:e8:ff
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:ce:41:c7:fb:b2:4f
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

root@network:~# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:00003a46b9a2f841
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 1(patch-int): addr:a2:83:a5:bd:48:2c
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(gre-0a0000c1): addr:4a:99:1c:39:50:06
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(snooper1): addr:6e:01:41:55:eb:49
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:3a:46:b9:a2:f8:41
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

I’ve the two required namespaces configured:

root@network:~# ip netns
qdhcp-b43d1b41-7123-4ffa-a1aa-d25853383bdb
qrouter-f3ba7133-39ea-4492-a746-d2e78c31593a

Dnsmasq binding seems right to me:

root@network:~# ps aux | grep dnsmasq
nobody    1714  0.0  0.1  28200   996 ?        S    13:26   0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap752ac585-ff --except-interface=lo --pid-file=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/host --addn-hosts=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b43d1b41-7123-4ffa-a1aa-d25853383bdb/opts --leasefile-ro --dhcp-authoritative --dhcp-range=set:tag0,192.168.1.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
root      6693  0.0  0.1  11740   932 pts/7    S+   14:14   0:00 grep --color=auto dnsmasq

Agent list seems also fine to me:

root@controller:~/download# neutron agent-list
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| id                                   | agent_type         | host     | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| 61db8227-c0d1-4802-950a-ae33e8f211a8 | Open vSwitch agent | network  | :-)   | True           | neutron-openvswitch-agent |
| 8a5d36f0-a72f-4f2a-917d-082a25c620bf | L3 agent           | network  | :-)   | True           | neutron-l3-agent          |
| a42998a5-12d3-4218-a2e2-36d1d59c49e8 | Metadata agent     | network  | :-)   | True           | neutron-metadata-agent    |
| c6658c6b-7283-4984-922b-08898741889c | DHCP agent         | network  | :-)   | True           | neutron-dhcp-agent        |
| ec2cd87b-bb19-42ea-874a-f885d3d2152f | Open vSwitch agent | compute1 | :-)   | True           | neutron-openvswitch-agent |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+

Any ideas?

Best regards, Dominik