Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

asked 2014-05-08 23:38:03 -0500

don gravatar image

DHCP request hits interface but not dnsmasq

I have a problem where DHCP broadcast hits the interface dnsmasq is listening on (according to tcpdump), but dnsmasq does not wake up (according to strace), so no response. the guest seems otherwise fine from the console, just no IP.

can anyone suggest where i should continue debugging this? I'm kind of running out of ideas. I was of the opinion that if tcpdump showed the packet hitting the interface, and iptables didn't say it dropped it, that dnsmasq should at least wake up and reject/answer.

System is ubuntu 14.04 using cloud-archive:icehouse.

To debug, i have disabled namespaces (and disabled allow_overlapping_ips). This is an all-in one, and i'm using local, ovs. ml2.

$ ps -ef|grep dnsm
nobody   10063     1  0 00:06 ?        00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tape2855db2-11 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host --addn-hosts=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts --leasefile-ro --dhcp-range=set:tag0,172.16.6.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

If i do tcpdump -i tape2855db2-11, I see:

IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a4:e5:a8

but no reply. strace on dnsmasq shows that the select() doesn't return. I have rp_filter=0, accept_local=0, forwarding=0 on sysctl.

$ sudo ifconfig tape2855db2-11
tape2855db2-11 Link encap:Ethernet  HWaddr fa:16:3e:3a:75:78  
              inet addr:172.16.6.2  Bcast:172.16.6.255  Mask:255.255.255.0
              inet6 addr: fe80::b8a5:3dff:fe34:8bb/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:76 errors:0 dropped:0 overruns:0 frame:0
              TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:22816 (22.8 KB)  TX bytes:1206 (1.2 KB)

$ sudo ovs-vsctl show 834141a3-07fc-4128-9770-62c7cfeba112 Bridge br-ex Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Bridge br-int Port "qvo65602f4f-bd" tag: 1 Interface "qvo65602f4f-bd" Port br-int Interface br-int type: internal Port "tape2855db2-11" tag: 1 Interface "tape2855db2-11" type: internal ovs_version: "2.0.1"

I don't see how if the packet hits that interface that dnsmasq wouldn't hear it? i tried adding logging to iptables:

*filter
:INPUT ACCEPT [142:34059]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:33939]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -j LOGGING
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A LOGGING -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
-A neutron-openvswi-sg-fallback -j DROP
-A nova-api-INPUT -d 127.0.1.1/32 -p tcp -m tcp --dport 8775 -j ACCEPT
-A nova-filter-top -j nova-api-local
COMMIT

and that doesn't show me dropping this packet.

the only errors i see in logs is this one:

Stderr: "device tap65602f4f-bd is already a member of a bridge; can't enslave it to bridge brq9fda8e70-72.\n"

which is repetitive, it happens ever few seconds in linuxbridge-agent.log. I don't know if its related, and i don't know why it happens.

in ml2_conf.ini, i have:

type_drivers = local,flat,vlan,gre,vxlan
mechanism_drivers = openvswitch,linuxbridge

in my install, i created the interface with:

    ovs-vsctl add-br br-int
    ifconfig br-int inet 172.16.6.1 netmask 255.255.255.0

$ ifconfig br-int
br-int    Link encap:Ethernet  HWaddr 02:af:d0:d2:7f:4a  
          inet addr:172.16.6.1  Bcast:172.16.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b41d:44ff:fe6c:76a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:125 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:33066 (33.0 KB)  TX bytes:648 (648.0 B)

DHCP request hits interface but not dnsmasq

I have a problem where DHCP broadcast hits the interface dnsmasq is listening on (according to tcpdump), but dnsmasq does not wake up (according to strace), so no response. the guest seems otherwise fine from the console, just no IP.

can anyone suggest where i should continue debugging this? I'm kind of running out of ideas. I was of the opinion that if tcpdump showed the packet hitting the interface, and iptables didn't say it dropped it, that dnsmasq should at least wake up and reject/answer.

System is ubuntu 14.04 using cloud-archive:icehouse.

To debug, i have disabled namespaces (and disabled allow_overlapping_ips). This is an all-in one, and i'm using local, ovs. ml2.

$ ps -ef|grep dnsm
nobody   10063     1  0 00:06 ?        00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tape2855db2-11 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host --addn-hosts=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts --leasefile-ro --dhcp-range=set:tag0,172.16.6.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

If i do tcpdump -i tape2855db2-11, I see:

IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a4:e5:a8

but no reply. strace on dnsmasq shows that the select() doesn't return. I have rp_filter=0, accept_local=0, forwarding=0 on sysctl.

$ sudo ifconfig tape2855db2-11
tape2855db2-11 Link encap:Ethernet  HWaddr fa:16:3e:3a:75:78  
              inet addr:172.16.6.2  Bcast:172.16.6.255  Mask:255.255.255.0
              inet6 addr: fe80::b8a5:3dff:fe34:8bb/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:76 errors:0 dropped:0 overruns:0 frame:0
              TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:22816 (22.8 KB)  TX bytes:1206 (1.2 KB)

$ sudo ovs-vsctl show 834141a3-07fc-4128-9770-62c7cfeba112 Bridge br-ex Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Bridge br-int Port "qvo65602f4f-bd" tag: 1 Interface "qvo65602f4f-bd" Port br-int Interface br-int type: internal Port "tape2855db2-11" tag: 1 Interface "tape2855db2-11" type: internal ovs_version: "2.0.1"

I don't see how if the packet hits that interface that dnsmasq wouldn't hear it? i tried adding logging to iptables:

*filter
:INPUT ACCEPT [142:34059]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:33939]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -j LOGGING
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A LOGGING -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
-A neutron-openvswi-sg-fallback -j DROP
-A nova-api-INPUT -d 127.0.1.1/32 -p tcp -m tcp --dport 8775 -j ACCEPT
-A nova-filter-top -j nova-api-local
COMMIT

and that doesn't show me dropping this packet.

the only errors i see in logs is this one:

Stderr: "device tap65602f4f-bd is already a member of a bridge; can't enslave it to bridge brq9fda8e70-72.\n"

which is repetitive, it happens ever few seconds in linuxbridge-agent.log. I don't know if its related, and i don't know why it happens.

in ml2_conf.ini, i have:

type_drivers = local,flat,vlan,gre,vxlan
mechanism_drivers = openvswitch,linuxbridge

in my install, i created the interface with:

    ovs-vsctl add-br br-int
    ifconfig br-int inet 172.16.6.1 netmask 255.255.255.0

$ ifconfig br-int
br-int    Link encap:Ethernet  HWaddr 02:af:d0:d2:7f:4a  
          inet addr:172.16.6.1  Bcast:172.16.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b41d:44ff:fe6c:76a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:125 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:33066 (33.0 KB)  TX bytes:648 (648.0 B)

I created the subnet as:

subnet-create --name "internal-subnet" --allocation-pool start=172.16.6.2,end=172.16.6.200 --gateway 172.16.6.11 internal 172.16.6.0/24

DHCP request hits interface but not dnsmasq

I have a problem where DHCP broadcast hits the interface dnsmasq is listening on (according to tcpdump), but dnsmasq does not wake up (according to strace), so no response. the guest seems otherwise fine from the console, just no IP.

can anyone suggest where i should continue debugging this? I'm kind of running out of ideas. I was of the opinion that if tcpdump showed the packet hitting the interface, and iptables didn't say it dropped it, that dnsmasq should at least wake up and reject/answer.

System is ubuntu 14.04 using cloud-archive:icehouse.

To debug, i have disabled namespaces (and disabled allow_overlapping_ips). This is an all-in one, and i'm using local, ovs. ml2.

$ ps -ef|grep dnsm
nobody   10063     1  0 00:06 ?        00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tape2855db2-11 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host --addn-hosts=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts --leasefile-ro --dhcp-range=set:tag0,172.16.6.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

If i do tcpdump -i tape2855db2-11, I see:

IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a4:e5:a8

but no reply. strace on dnsmasq shows that the select() doesn't return. I have rp_filter=0, accept_local=0, forwarding=0 on sysctl.

$ sudo ifconfig tape2855db2-11
tape2855db2-11 Link encap:Ethernet  HWaddr fa:16:3e:3a:75:78  
              inet addr:172.16.6.2  Bcast:172.16.6.255  Mask:255.255.255.0
              inet6 addr: fe80::b8a5:3dff:fe34:8bb/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:76 errors:0 dropped:0 overruns:0 frame:0
              TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:22816 (22.8 KB)  TX bytes:1206 (1.2 KB)

$ sudo ovs-vsctl show 834141a3-07fc-4128-9770-62c7cfeba112 Bridge br-ex Port "eth1" Interface "eth1" Port br-ex Interface br-ex type: internal Bridge br-int Port "qvo65602f4f-bd" tag: 1 Interface "qvo65602f4f-bd" Port br-int Interface br-int type: internal Port "tape2855db2-11" tag: 1 Interface "tape2855db2-11" type: internal ovs_version: "2.0.1"

"2.0.1"

I don't see how if the packet hits that interface that dnsmasq wouldn't hear it? i tried adding logging to iptables:

*filter
:INPUT ACCEPT [142:34059]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:33939]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -j LOGGING
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A LOGGING -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
-A neutron-openvswi-sg-fallback -j DROP
-A nova-api-INPUT -d 127.0.1.1/32 -p tcp -m tcp --dport 8775 -j ACCEPT
-A nova-filter-top -j nova-api-local
COMMIT

and that doesn't show me dropping this packet.

the only errors i see in logs is this one:

Stderr: "device tap65602f4f-bd is already a member of a bridge; can't enslave it to bridge brq9fda8e70-72.\n"

which is repetitive, it happens ever few seconds in linuxbridge-agent.log. I don't know if its related, and i don't know why it happens.

in ml2_conf.ini, i have:

type_drivers = local,flat,vlan,gre,vxlan
mechanism_drivers = openvswitch,linuxbridge

in my install, i created the interface with:

    ovs-vsctl add-br br-int
    ifconfig br-int inet 172.16.6.1 netmask 255.255.255.0

$ ifconfig br-int
br-int    Link encap:Ethernet  HWaddr 02:af:d0:d2:7f:4a  
          inet addr:172.16.6.1  Bcast:172.16.6.255  Mask:255.255.255.0
          inet6 addr: fe80::b41d:44ff:fe6c:76a/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:125 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:33066 (33.0 KB)  TX bytes:648 (648.0 B)

I created the subnet as:

subnet-create --name "internal-subnet" --allocation-pool start=172.16.6.2,end=172.16.6.200 --gateway 172.16.6.11 internal 172.16.6.0/24