DHCP request hits interface but not dnsmasq
I have a problem where DHCP broadcast hits the interface dnsmasq is listening on (according to tcpdump), but dnsmasq does not wake up (according to strace), so no response. the guest seems otherwise fine from the console, just no IP.
can anyone suggest where i should continue debugging this? I'm kind of running out of ideas. I was of the opinion that if tcpdump showed the packet hitting the interface, and iptables didn't say it dropped it, that dnsmasq should at least wake up and reject/answer.
System is ubuntu 14.04 using cloud-archive:icehouse.
To debug, i have disabled namespaces (and disabled allow_overlapping_ips). This is an all-in one, and i'm using local, ovs. ml2.
$ ps -ef|grep dnsm
nobody 10063 1 0 00:06 ? 00:00:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tape2855db2-11 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/host --addn-hosts=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts --leasefile-ro --dhcp-range=set:tag0,172.16.6.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
If i do tcpdump -i tape2855db2-11, I see:
IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a4:e5:a8
but no reply. strace on dnsmasq shows that the select() doesn't return. I have rp_filter=0, accept_local=0, forwarding=0 on sysctl.
$ sudo ifconfig tape2855db2-11
tape2855db2-11 Link encap:Ethernet HWaddr fa:16:3e:3a:75:78
inet addr:172.16.6.2 Bcast:172.16.6.255 Mask:255.255.255.0
inet6 addr: fe80::b8a5:3dff:fe34:8bb/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:76 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:22816 (22.8 KB) TX bytes:1206 (1.2 KB)
$ sudo ovs-vsctl show
834141a3-07fc-4128-9770-62c7cfeba112
Bridge br-ex
Port "eth1"
Interface "eth1"
Port br-ex
Interface br-ex
type: internal
Bridge br-int
Port "qvo65602f4f-bd"
tag: 1
Interface "qvo65602f4f-bd"
Port br-int
Interface br-int
type: internal
Port "tape2855db2-11"
tag: 1
Interface "tape2855db2-11"
type: internal
ovs_version: "2.0.1"
I don't see how if the packet hits that interface that dnsmasq wouldn't hear it? i tried adding logging to iptables:
*filter
:INPUT ACCEPT [142:34059]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [136:33939]
:LOGGING - [0:0]
:neutron-filter-top - [0:0]
:neutron-openvswi-FORWARD - [0:0]
:neutron-openvswi-INPUT - [0:0]
:neutron-openvswi-OUTPUT - [0:0]
:neutron-openvswi-local - [0:0]
:neutron-openvswi-sg-chain - [0:0]
:neutron-openvswi-sg-fallback - [0:0]
:nova-api-FORWARD - [0:0]
:nova-api-INPUT - [0:0]
:nova-api-OUTPUT - [0:0]
:nova-api-local - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j neutron-openvswi-INPUT
-A INPUT -j nova-api-INPUT
-A INPUT -j LOGGING
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-api-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-api-OUTPUT
-A neutron-filter-top -j neutron-openvswi-local
-A LOGGING -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
-A ...
check the dhcp-agent log and that
/var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/hosthas entries. Also there should be dhsmasq messages in the syslog.there is nothing of note in the dhcp-agent log. as i noted, the dnsmasq (even when started by hand) is not waking up when the dhcp packet arrives on the interface.
the /var/lib/.... does have host entry with mac fwiw.
in the ipfilter chain, i get to here(1) and here(2) in the below. If i run dnsmasq manually and i let it go to all interfaces (remove the bind interface), it does receive the dhcp (but of course doesn't answer since it comes from the other bridge ports that don't have IP).
(more)Please, enable namespaces and for qdhcp-network-id namespace corresponding your private network,which is in trouble execute commands
ip netns exec qdhcp-network-id ifconfig
ip netns exec qdhcp-network-id tcpdump -ln -i tap-interface (reported by previous run of ifconfig)
ip netns exec qdhcp-network-id netstat -lntp | grep dnsmasq
Another step to troubleshoot to compare
$ cat /var/lib/neutron/dhcp/9fda8e70-7293-4f29-a485-674e07acfdbc/opts
and
$ip netns exec qdhcp-network-id route -n
They should point the same router in opts file and gateway ( within namespace)
Could you format ovs-vsctl show output ? To make possible to read this report and understand your issues.