Ask Your Question
2

troubleshooting neutron - unreachable instance

asked 2014-05-29 15:57:12 -0500

ian.s.mcb gravatar image

Could someone share with me some advice on how to troubleshoot a broken neutron config?

I have an instance that is unreachable outside of the "qdhcp" network namespace, even though it has been assigned a floating IP address on an external network. Also, when I do a "ip a" on the "qrouter" network namespace, only a loopback device is displayed.

I am using the Icehouse release with the GRE ML2 plugin for Neutron. My setup consists of two nodes (one controller/network node and one compute node), and I used RDO/packstack to install everything.

edit retag flag offensive close merge delete

Comments

You wrote: Also, when I do a "ip a" on the "qrouter" network namespace, only a loopback device is displayed. In case you missing qg-xxxxxxxxxxx intreface your VMs are unreachable via floating IP. Output on my controller :-

[root@icehouse1 ~(keystone_admin)]# ip netns exec qrouter-2462467b-ea0a-4a40-a093-493572010694 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
32: qr-bbba6fd3-a3: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:84:1d:b6 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-bbba6fd3-a3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe84:1db6/64 scope ...
(more)
dbaxps gravatar imagedbaxps ( 2014-05-29 21:56:09 -0500 )edit

Would try to recreate router and private && external networks

dbaxps gravatar imagedbaxps ( 2014-05-29 21:58:01 -0500 )edit

Thanks for sharing your output. Now I now that I am missing the "qr" and "qg" devices within my "qrouter" namespace.

ian.s.mcb gravatar imageian.s.mcb ( 2014-05-30 15:47:12 -0500 )edit

Missing qr-* && qg-* interfaces , you cannot have qrouter namespace NAT working for you. I mean this :-

[root@icehouse1 ~(keystone_admin)]# ip netns exec qrouter-2462467b-ea0a-4a40-a093-493572010694 iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.1.96/32 -j DNAT --to-destination 10.0.0.68
-A neutron-l3-agent-OUTPUT -d 192.168.1.97/32 -j DNAT --to-destination 10.0.0.69
-A neutron-l3-agent-OUTPUT -d 192.168.1.100/32 -j DNAT --to-destination 10.0.0.73
-A neutron-l3-agent-OUTPUT -d 192.168.1.98/32 -j DNAT --to-destination 10.0.0.70
-A neutron-l3-agent-OUTPUT -d 192.168.1.99/32 -j DNAT --to-destination 10.0.0.72
-A neutron-l3-agent-OUTPUT -d 192.168.1.92/32 ...
(more)
dbaxps gravatar imagedbaxps ( 2014-05-30 22:05:49 -0500 )edit

For Two Node Neutron ML2+GRE+OVS setup , there is a sample of *.conf && *.ini files
http://kashyapc.fedorapeople.org/virt...

dbaxps gravatar imagedbaxps ( 2014-05-30 23:47:32 -0500 )edit

There were a lot of differences between your .conf/ .ini sample files and my ones. My ones came straight from running packstack. Of course, I also applied the steps explained here to get ML2: http://openstack.redhat.com/ML2_plugin

For now, I am going to stick with my files because I am uncertain about your .conf/.ini sample files.

ian.s.mcb gravatar imageian.s.mcb ( 2014-06-01 13:39:44 -0500 )edit

I rebooted my nodes without changing any config files and now I somehow have the qr-* and qg-* interfaces on the qrouter-* namespace. Nice!

Unfortunately, now my instances are completely unreachable, even from within the qdhcp-* namespace. Any suggestions?

Running "ip netns exec qdhcp-* ip a" shows a tap* interface and a loopback interface. No eth0 interface at all.

ian.s.mcb gravatar imageian.s.mcb ( 2014-06-01 13:45:33 -0500 )edit

My outputs (qdhcp-* namespace troubleshooting)

[root@icehouse1 ~(keystone_admin)]# ip netns exec qdhcp-a2bf6363-6447-47f5-a243-b998d206d593 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
16: tapa7e1ac48-7b: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether fa:16:3e:9d:87:4d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/24 brd 10.0.0.255 scope global tapa7e1ac48-7b
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe9d:874d/64 scope link 
       valid_lft forever preferred_lft forever
[root@icehouse1 ~(keystone_admin)]# ip netns exec qdhcp-a2bf6363-6447-47f5-a243-b998d206d593 netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp ...
(more)
dbaxps gravatar imagedbaxps ( 2014-06-01 21:23:28 -0500 )edit
  1. View also /var/log/neutron/dnsmasq.log for DHCPREQUESTS&&DHCPACKS

    1. Start VM and run

      ip netns exec qdhcp-a2bf6363-6447-47f5-a243-b998d206d593 tcpdump -ln -i tapa7e1ac48-7b

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tapa7e1ac48-7b, link-type EN10MB (Ethernet), capture size 65535 bytes 06:28:05.565315 ARP, Request who-has 10.0.0.73 tell 10.0.0.1, length 28 06:28:06.565277 ARP, Request who-has 10.0.0.73 tell 10.0.0.1, length 28 06:28:07.567277 ARP, Request who-has 10.0.0.73 tell 10.0.0.1, length 28
    06:28:08.088496 IP 10.0.0.72.bootpc > 10.0.0.11.bootps: BOOTP/DHCP, Request from fa:16:3e:7a:2b:ce, length 300
    06:28:08.089299 IP 10.0.0.11.bootps > 10.0.0.72.bootpc: BOOTP/DHCP, Reply, length 324

dbaxps gravatar imagedbaxps ( 2014-06-01 21:25:27 -0500 )edit

Here's the output from the commands that you gave me.

$ ip netns exec qdhcp-05d575e6-f98e-41d1-b0c9-09ab9274f009 ip a
12: tap9d55e069-3a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:13:75:18 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global tap9d55e069-3a
    inet6 fe80::f816:3eff:fe13:7518/64 scope link 
       valid_lft forever preferred_lft forever
13: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

$ ip netns exec qdhcp-05d575e6-f98e-41d1-b0c9-09ab9274f009 netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 10.0.0.3:53                 0.0.0.0:*                   LISTEN ...
(more)
ian.s.mcb gravatar imageian.s.mcb ( 2014-06-02 09:52:09 -0500 )edit

This is the tcpdump output that I get when pinging google from within the instance:

$ ip netns exec qdhcp-05d575e6-f98e-41d1-b0c9-09ab9274f009 tcpdump -ln -i tap9d55e069-3a
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap9d55e069-3a, link-type EN10MB (Ethernet), capture size 65535 bytes
10:33:50.657112 IP 10.0.0.6.44805 > 10.0.0.3.domain: 2+ A? google.com. (28)
10:33:50.657271 IP 10.0.0.3.domain > 10.0.0.6.44805: 2 Refused 0/0/0 (28)
10:33:50.658010 IP 10.0.0.6.34552 > 10.0.0.3.domain: 3+ A? google.com. (28)
10:33:50.658118 IP 10.0.0.3.domain > 10.0.0.6.34552: 3 Refused 0/0/0 (28)
10:33:50.658654 IP 10.0.0.6.60201 > 10.0.0.3.domain: 4+ A? google.com. (28)
10:33:50 ...
(more)
ian.s.mcb gravatar imageian.s.mcb ( 2014-06-02 09:58:00 -0500 )edit

Start your VM and run :-
$ip netns exec qdhcp-05d575e6-f98e-41d1-b0c9-09ab9274f009 tcpdump -ln -i tap9d55e069-3a
for 10-15 min
Temporary set dhcp_lease_time=120 in nova.conf

dbaxps gravatar imagedbaxps ( 2014-06-02 10:03:49 -0500 )edit

I added "dhcp_lease_time=120" to nova.conf. After setting up the tcpdump process, should I do anything from within the instance? And what output should I expect from tcpdump?

ian.s.mcb gravatar imageian.s.mcb ( 2014-06-02 11:18:26 -0500 )edit

Restart Nova Services. Activate tcpdump as shown , just start VM and keep running .Watch packages capturing at tap interface. You are supossed to see DHCP requests and replies every 2 min, if instance (VM) is getting IP via DHCP requests to DNSMASQ. If it doesn't happen VM is not getting private IPs from DNSMASQ. You will see everything on the screen.

dbaxps gravatar imagedbaxps ( 2014-06-02 11:30:29 -0500 )edit

No network traffic was displayed. How do you recommend troubleshooting DNSMASQ?

ian.s.mcb gravatar imageian.s.mcb ( 2014-06-02 13:50:04 -0500 )edit

Check whether it's running
$ps -ef | grep dnsmasq
If yes see /var/log/neutron/dnsmasq.log

Sample 
[root@icehouse1 ~(keystone_admin)]# ps -ef | grep dnsmasq
nobody    3986     1  0 05:57 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap15ef9839-d1 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/b269f328-ae97-48fd-85f0-5ed02f8532f4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b269f328-ae97-48fd-85f0-5ed02f8532f4/host --addn-hosts=/var/lib/neutron/dhcp/b269f328-ae97-48fd-85f0-5ed02f8532f4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b269f328-ae97-48fd-85f0-5ed02f8532f4/opts --leasefile-ro --dhcp-range=set:tag0,40.0.0.0,static,120s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
nobody    3989     1  0 05:57 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapa7e1ac48-7b --except-interface=lo --pid-file=/var/lib/neutron/dhcp/a2bf6363-6447-47f5-a243-b998d206d593/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/a2bf6363-6447-47f5-a243-b998d206d593/host --addn-hosts=/var/lib/neutron/dhcp/a2bf6363-6447-47f5-a243-b998d206d593/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/a2bf6363-6447-47f5-a243-b998d206d593/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
dbaxps gravatar imagedbaxps ( 2014-06-02 13:57:45 -0500 )edit

One more test : start your VM and try :-
$ ip netns exec qdhcp-05d575e6-f98e-41d1-b0c9-09ab9274f009 ping -c 5 10.0.0.6

dbaxps gravatar imagedbaxps ( 2014-06-02 14:06:40 -0500 )edit

Thanks for the help dbaxps! I ended up resolving my problem by reverting to nova-network, which is a lot easier to understand.

ian.s.mcb gravatar imageian.s.mcb ( 2014-06-26 11:30:28 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2014-05-29 22:15:04 -0500

annegentle gravatar image

This section should be useful.

http://docs.openstack.org/trunk/opens...

edit flag offensive delete link more

Comments

This has the information I was looking for.

ian.s.mcb gravatar imageian.s.mcb ( 2014-05-30 15:47:39 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-05-29 15:57:12 -0500

Seen: 1,560 times

Last updated: May 29 '14