Ask Your Question

Neutron network node: DHCP requests hit interface, but not dnsmasq for some tenants while ok for others

asked 2015-04-13 01:37:56 -0500

Tomas gravatar image

updated 2015-04-15 09:28:21 -0500

I am struggling with network node Neutron implementation not giving IP addresses in certain tenants (after controller reboot some tenants start to work while other stop to, looks like issue is not related to any particular setup in tenant).

In both cases I see DHCP requests arriving on tap interface, dnsmasq process running correctly in tenant, but in case of some tenants DHCP replies are not seen and it seems nothing is delivered to dnsmasq process. Restarting neutron-dhcp-agent nor killing dnsmasq proces (and then restarting neutron-dhcp-agent) does not help.

I am out of ideas...any help greatly appreciated !

Not working tenant:

ip netns exec qdhcp-363efdba-9c30-4305-8a6a-7639a66a13fd ip a
190: tap51abb5e4-c1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default

ip netns exec qdhcp-363efdba-9c30-4305-8a6a-7639a66a13fd tcpdump port 67 or port 68 -i tap51abb5e4-c1 -nnvvvNXs 512

... I see DHCP requests, but no DHCP reply

ps -efa | grep tap51abb5e4-c1
nobody   23933     1  0 08:00 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap51abb5e4-c1 --except-interface=lo --pid-file=/var/run/neutron/dhcp/363efdba-9c30-4305-8a6a-7639a66a13fd/pid --dhcp-hostsfile=/var/run/neutron/dhcp/363efdba-9c30-4305-8a6a-7639a66a13fd/host --addn-hosts=/var/run/neutron/dhcp/363efdba-9c30-4305-8a6a-7639a66a13fd/addn_hosts --dhcp-optsfile=/var/run/neutron/dhcp/363efdba-9c30-4305-8a6a-7639a66a13fd/opts --leasefile-ro --dhcp-range=set:tag0,,static,172800s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq/dnsmasq-neutron.conf --domain=openstacklocal

Looking at process, no packets are arriving strace -p 23933 -e network,write -s 4096

I have noticed there is no tag in OVS assigned (example of 2 not-working networks):

    Port "tap51abb5e4-c1"
        Interface "tap51abb5e4-c1"
            type: internal

    Port "tap5d9b6dc3-32"
        Interface "tap5d9b6dc3-32"
            type: internal

Doing the same for working tenant:

I see request and offer + packets reported on process via strace -p 24505 -e network,write -s 4096

Process 24505 attached
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(68), sin_addr=inet_addr("")}, msg_iov(1)=[{"\1\1\6\0\0022\341M\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\372\26>\37.\211\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 ...
edit retag flag offensive close merge delete


search the ovs agent logs for these taps to find why they don't have a tag

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-04-15 01:30:00 -0500 )edit

Thanks !

I can confirm tags are causing this problem. In logs I can see ovs-vsctl command that creates tap interfaces for all four. But later for woring ones I get:

/usr/bin/ovs-vsctl --timeout=10 set Port tapb8d8b645-21 tag=10

How is tag defined and when is configured by what? Where to look?

Tomas gravatar imageTomas ( 2015-04-15 06:28:32 -0500 )edit

search the neutron logs (server and ovs agent) for the ids (eg 51abb5e4) to see if they say why the binding fails. I believe there is a bug where if the port binding fails once, then the port is unbindable for ever

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-04-15 06:45:52 -0500 )edit

With the admin credentials sourced, can you provide the output of neutron port-show <DHCP-PORT-ID>. You can find DHCP-PORT-ID by running neutron port-list --device_owner network:dhcp

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-04-15 08:15:32 -0500 )edit

I have edited question and added port-show for all dhcp ports in my environment (needed to ad as text as upload was not available for me). That includes some working, some not - I have not spotted differences here.

Thanks for your help so far

Tomas gravatar imageTomas ( 2015-04-15 09:31:04 -0500 )edit

1 answer

Sort by ยป oldest newest most voted

answered 2015-04-15 01:24:08 -0500

If restarting neutron-openvswitch-agent and corresponding dnsmasq daemons won't help, you would have recreate both of private networks.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2015-04-12 12:48:24 -0500

Seen: 1,493 times

Last updated: Apr 15 '15