Ask Your Question
2

dns not working and dhcp namespace has two tap interfaces

asked 2015-01-14 18:22:35 -0500

T u l gravatar image

updated 2015-01-17 02:02:18 -0500

darragh-oreilly gravatar image

Hi,

A problem appeared (I guess) when I tried to add another external network to add one more floating ip range, and then deleted it.

Now, dns does not reply in my vms (for some tenants it does) What i found out is that in my qdhcp namespace i got two tap interfaces,

     ip netns exec qdhcp-43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea ifconfig
...
tap2e4ca852-ea: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.3  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::f816:3eff:fe04:1659  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:04:16:59  txqueuelen 0  (Ethernet)
        RX packets 148263  bytes 20487809 (19.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25210  bytes 3433433 (3.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tapfaa2eb73-9f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.17  netmask 255.255.255.0  broadcast 192.168.33.255
        inet6 fe80::f816:3eff:fed0:e2c  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:d0:0e:2c  txqueuelen 0  (Ethernet)
        RX packets 429  bytes 40643 (39.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 118  bytes 10171 (9.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Moreover, when i do nslookup from a VM i get packets only on one tap interface: tap2e4ca852-ea

In VM's resov.conf, i have two nameservers: 192.168.33.3 and 192.168.33.17 but traffic from both of them go to one tap interface....

however, in my process list i got 

nobody    2236  0.0  0.0  15528   756 ?        S    22:38   0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapfaa2eb73-9f --except-interface=lo --pid-file=/var/lib/neutron/dhcp/43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea/host --addn-hosts=/var/lib/neutron/dhcp/43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea/opts --leasefile-ro --dhcp-range=set:tag0,192.168.33.0,static,86400s --dhcp-lease-max=256 --conf-file= --server=10.21.0.4 --domain=uni.lux

as you see, --interface=tapfaa2eb73-9f is the interface where packets do not simply go.

I also went to /var/lib/neutron/dhcp/43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea and tried to change the content of interface file from tapfaa2eb73-9f to tap2e4ca852-ea but after restarting neutron-dhcp-agent, the value of that file was restored.

Question: how do I change the tap interface to bind dnsmasq? (or maybe how to make traffic go to that interface dnsmasq is bond to)?

I have an RDO installation IceHouse on Fedora 20; VLAN for tenants.

Thanks for any hints!

After darragh-oreilly's suggestions.

I removed port that started with faa2eb73-9f

neutron port-delete faa2eb73-9f8f-4e66-a50a-846e8f283636

restarted all neutron services on controller/network node; also restarted openvswitch

What changed? in VM's resolv.conf i get now 1 nameserver nameserver 192.168.33.21

TAP interfaces: new one appeared; old stayed ip netns exec qdhcp-43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea ifconfig

tap05de60a7-45: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.33.21 ...
(more)
edit retag flag offensive close merge delete

Comments

I'm not sure how you got to this situation, but it seems like a bug that this network has two ports for this host. Do neutron port-list to get the full port-ids and neutron port-show $PORT_ID on each port. Maybe deleting the port that starts with faa2eb73-9f and restarting the agent will do it.

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-01-15 11:51:12 -0500 )edit

I deleted a port that started with faa2eb.. and restarted dhcp-agent. Now i have 3 tap interfaces in the same namespace. The faa2eb is still there...

T u l gravatar imageT u l ( 2015-01-15 15:22:54 -0500 )edit

who should kill these tap interfaces when port is deleted?

T u l gravatar imageT u l ( 2015-01-15 16:47:12 -0500 )edit

so you deleted it with neutron port-delete $PORT_ID ? Can you post neutron port-show $PORT_ID for the 3 ports - as admin so the port:bindings etc are shown?

darragh-oreilly gravatar imagedarragh-oreilly ( 2015-01-16 01:10:30 -0500 )edit

Thanks for helping out! I updated it in the question (now it's long...) Should I also delete the 1st (oldest) port 2e4ca852-eaca-45f6-ad2a-06f6fa6f8601?

T u l gravatar imageT u l ( 2015-01-16 02:29:13 -0500 )edit
see more comments

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-01-16 09:13:23 -0500

darragh-oreilly gravatar image

updated 2015-01-17 01:58:14 -0500

Not an answer ...

The agent is not reusing the dhcp tap ports because it is seeing a different device_id each time - you will need to figure out why. Try this:

$ ipython
In [1]: from neutron.common.utils import get_dhcp_agent_device_id

In [2]: print get_dhcp_agent_device_id("43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea", "celeste.uni.lux")
dhcp6e441d8c-18d7-59f4-bca1-5efd245c8cb9-43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea

In [6]: print get_dhcp_agent_device_id("43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea", "celeste")
dhcp6e441d8c-18d7-59f4-bca1-5efd245c8cb9-43bf6a7d-8b0a-4be2-90dc-4f75de1ee4ea

With debug on, what does grep " host *=" $NEUTRON_DHCP_LOG say? Is there anything peculiar about your machine naming system?

update: The way the device_id is generated was changed in a patch which was backported to the stable icehouse branch https://github.com/openstack/neutron/...

With this patch, only hostname is used, while before hostname.domainname was used.

Anyway you don't have the patch and you changed the domainname:

In [1]: import uuid

In [2]: uuid.uuid5(uuid.NAMESPACE_DNS, "celeste.uni.lu")
Out[2]: UUID('dcc7fda7-d231-5cec-a3a7-37a8d69a96cc')

In [3]: uuid.uuid5(uuid.NAMESPACE_DNS, "celeste.uni.lux")
Out[3]: UUID('8d35e02c-de0f-53c4-b95b-ac8a8b3d2c39')

and these are the numbers you see in the device_id.

So I think you should neutron port-delete any ports with the old name and remove their taps from the dhcp namespace (or reboot the netnode).

edit flag offensive delete link more

Comments

Thanks, but in my version neutron.common.utils does not have get_dhcp_agent_device_id. a part of dir():

 'cache_method_results',
 'cfg',
 'compare_elements',
 'dict2str',
 'dict2tuple',
 'diff_list_of_dict',
 'find_config_file',
 'functools',
T u l gravatar imageT u l ( 2015-01-16 12:56:00 -0500 )edit

grep " host *=" /var/log/neutron/dhcp-agent.log

2015-01-16 20:11:57.981 15625 DEBUG neutron.openstack.common.service [req-1a029cdc-5fd5-4c97-ad26-2b6d4c2bc33f None] host                           = celeste.uni.lux log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1945
T u l gravatar imageT u l ( 2015-01-16 13:12:09 -0500 )edit

host celeste

celeste.uni.lux has address 10.41.0.82

host celeste.uni.lux

celeste.uni.lux has address 10.41.0.82

host celeste.uni.lu

Host celeste.uni.lu not found: 3(NXDOMAIN)

ping celeste.uni.lu

PING celeste.uni.lu (10.41.0.82) 56(84) bytes of data.

...

T u l gravatar imageT u l ( 2015-01-16 13:15:16 -0500 )edit

the hostname was celeste.uni.lu, but it was not resolving. I though it caused some problems and recently changed to celeste.uni.lux; also added celeste.uni.lu to /etc/hosts

T u l gravatar imageT u l ( 2015-01-16 13:19:11 -0500 )edit

I guess the patch is not in RDO yet: yum install openstack-neutron

Loaded plugins: langpacks, priorities
openstack-icehouse
...
232 packages excluded due to repository priority protections
Package openstack-neutron-2014.1.3-5.fc21.noarch already installed and latest version
T u l gravatar imageT u l ( 2015-01-17 07:52:49 -0500 )edit
see more comments

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

subscribe to rss feed

Stats

Asked: 2015-01-14 18:22:35 -0500

Seen: 1,230 times

Last updated: Jan 17 '15

Related questions

Upgrading RDO To Icehouse on Fedora 20

Neutron Flat networking with no dhcp [closed]

l2 links between VMs

Adding new allocation pool to existing network

PackStack installation networking issue

RDO neutron on github

Unable to delete neutron port [closed]

What other steps should I take with networking after adding a compute node?

Network is slow on Compute Node of Two Node IceHouse Neutron OVS&VLAN CentOS 6.5 (Fedora 20) Cluster

Lbaas in Kilo error