Ask Your Question
1

DHCP request hits interface but not dnsmasq

asked 2014-05-08 23:38:03 -0500

don gravatar image

updated 2014-05-10 11:55:09 -0500

I have a problem where DHCP broadcast hits the interface dnsmasq is listening on (according to tcpdump), but dnsmasq does not wake up (according to strace), so no response. the guest seems otherwise fine from the console, just no IP.

can anyone suggest where i should continue debugging this? I'm kind of running out of ideas. I was of the opinion that if tcpdump showed the packet hitting the interface, and iptables didn't say it dropped it, that dnsmasq should at least wake up and reject/answer.

System is ubuntu 14.04 using cloud-archive:icehouse.

To debug, i have disabled namespaces (and disabled allow_overlapping_ips). This is an all-in one, and i'm using local, ovs. ml2.

$ ps -ef|grep dnsm
nobody   10063     1  0 00:06 ?        00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tape2855db2-11 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host --addn-hosts=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts --leasefile-ro --dhcp-range=set:tag0,172.16.6.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

If i do tcpdump -i tape2855db2-11, I see:

IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a4:e5:a8

but no reply. strace on dnsmasq shows that the select() doesn't return. I have rp_filter=0, accept_local=0, forwarding=0 on sysctl.

$ sudo ifconfig tape2855db2-11
tape2855db2-11 Link encap:Ethernet  HWaddr fa:16:3e:3a:75:78  
              inet addr:172.16.6.2  Bcast:172.16.6.255  Mask:255.255.255.0
              inet6 addr: fe80::b8a5:3dff:fe34:8bb/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:76 errors:0 dropped:0 overruns:0 frame:0
              TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:22816 (22.8 KB)  TX bytes:1206 (1.2 KB)

$ sudo ovs-vsctl show 
    834141a3-07fc-4128-9770-62c7cfeba112
        Bridge br-ex
            Port "eth1"
                Interface "eth1"
            Port br-ex
                Interface br-ex
                    type: internal
        Bridge br-int
            Port "qvo65602f4f-bd"
                tag: 1
                Interface "qvo65602f4f-bd"
            Port br-int
                Interface br-int
                    type: internal
            Port "tape2855db2-11"
                tag: 1
                Interface "tape2855db2-11"
                    type: internal
        ovs_version: "2.0.1"

I don't see how if the packet hits that interface that dnsmasq wouldn't hear it? i tried adding logging to iptables:

*filter
:INPUT ACCEPT [142:34059]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:33939]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -j LOGGING
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A LOGGING -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
-A ...
(more)
edit retag flag offensive close merge delete

Comments

check the dhcp-agent log and that /var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host has entries. Also there should be dhsmasq messages in the syslog.

darragh-oreilly gravatar imagedarragh-oreilly ( 2014-05-09 02:40:47 -0500 )edit

there is nothing of note in the dhcp-agent log. as i noted, the dnsmasq (even when started by hand) is not waking up when the dhcp packet arrives on the interface.

the /var/lib/.... does have host entry with mac fwiw.

don gravatar imagedon ( 2014-05-09 13:26:07 -0500 )edit

in the ipfilter chain, i get to here(1) and here(2) in the below. If i run dnsmasq manually and i let it go to all interfaces (remove the bind interface), it does receive the dhcp (but of course doesn't answer since it comes from the other bridge ports that don't have IP).

*filter
:INPUT ACCEPT [427:133276]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [423:134101]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-i2b86bd86-5 - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-o2b86bd86-5 - [0:0]
:neutron-openvswi-s2b86bd86-5 - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j LOG -p udp --dport 67 --log-prefix "here(1) "
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD ...
(more)
don gravatar imagedon ( 2014-05-09 14:45:23 -0500 )edit

Please, enable namespaces and for qdhcp-network-id namespace corresponding your private network,which is in trouble execute commands
ip netns exec qdhcp-network-id ifconfig
ip netns exec qdhcp-network-id tcpdump -ln -i tap-interface (reported by previous run of ifconfig)
ip netns exec qdhcp-network-id netstat -lntp | grep dnsmasq
Another step to troubleshoot to compare
$ cat /var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts
and
$ip netns exec qdhcp-network-id route -n
They should point the same router in opts file and gateway ( within namespace)

DanIzack gravatar imageDanIzack ( 2014-05-10 04:44:39 -0500 )edit

Could you format ovs-vsctl show output ? To make possible to read this report and understand your issues.

DanIzack gravatar imageDanIzack ( 2014-05-10 05:17:12 -0500 )edit

2 answers

Sort by » oldest newest most voted
1

answered 2014-05-10 13:29:02 -0500

don gravatar image

I had an IP address on the br-int interface (on the DHCP subnet). This caused the problem above in the use_namespaces case (where the DHCP request came in and was responded, but did not arrive @ the guest).

Removing the IP from the br-int interface solved that problem.

edit flag offensive delete link more
2

answered 2014-05-11 07:29:34 -0500

DanIzack gravatar image
The core troubleshooting steps  brought Don to a right decision were identifying the name of internal interface within corresponding qdhcp namespace and  capturing packages on internal qdhcp interface tap07519b2e-d6 =>
$ sudo ip netns exec qdhcp-bcb826f9-545a-4d32-ba94-8f9fb71054bc ifconfig
$ sudo ip netns exec qdhcp-bcb826f9-545a-4d32-ba94-8f9fb71054bc tcpdump -ln -i tap07519b2e-d6 
It did provide him information, that dnsmasq daemon  was providing DHCP replays  (ACKs ) to  guest VM, however replays were not accepted by guest due to incorrect br-int configuration.
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-05-08 23:38:03 -0500

Seen: 7,184 times

Last updated: May 11 '14