Ask Your Question
0

nova-network not forwarding properly from VM->public

asked 2011-08-24 17:22:57 -0500

Hey there,

I currently have a 3 node Openstack configuration using Vlan networking. One host (os-controller) runs Nova-network/Nova-scheduler and Nova-API. The other two hosts run compute and volume.

This setup was working as expected until we had to reboot the controller on Monday. After that happened, the following odd traffic patterns were noted.

Floating IPs bound to instances worked as expected. Fixed IPs would reply from the IP of the controller. i.e., if I opened up an SSH session, I could see the connection from my workstation make it to the VM and I could see the ssh banner response from the VM -> OS controllers vlan interface. However when the OS-Controller replied, my workstation would see the response as coming from the controller and not the IP of the instance (via tcpdump -A -s 65000 and looking for the SSH response banner in the payload).

So the situation now is that when I try to SSH to the IP of a VM, traffic goes to 10.1.100.11 (for example), but responses come back from 10.1.0.76.

I've put what I believe to be the relevant info in the paste at http://paste.openstack.org/show/2263/ .

Thank you for your help. :)

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted
0

answered 2011-08-24 18:04:09 -0500

iptables-save output.

Generated by iptables-save v1.4.4 on Wed Aug 24 13:03:30 2011

*nat :PREROUTING ACCEPT [45899:4908527] :POSTROUTING ACCEPT [523:32419] :OUTPUT ACCEPT [396:27954] :nova-network-OUTPUT - [0:0] :nova-network-POSTROUTING - [0:0] :nova-network-PREROUTING - [0:0] :nova-network-floating-snat - [0:0] :nova-network-snat - [0:0] :nova-postrouting-bottom - [0:0] -A PREROUTING -j nova-network-PREROUTING -A POSTROUTING -j nova-network-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A OUTPUT -j nova-network-OUTPUT -A nova-network-OUTPUT -d 10.1.0.89/32 -j DNAT --to-destination 10.1.100.8 -A nova-network-OUTPUT -d 10.1.0.84/32 -j DNAT --to-destination 10.1.100.9 -A nova-network-OUTPUT -d 10.1.0.85/32 -j DNAT --to-destination 10.1.100.10 -A nova-network-POSTROUTING -s 10.1.100.0/22 -d 10.128.0.0/24 -j ACCEPT -A nova-network-POSTROUTING -s 10.1.100.0/22 -d 10.1.100.0/22 -j ACCEPT -A nova-network-PREROUTING -d 10.1.0.76/32 -p udp -m udp --dport 1000 -j DNAT --to-destination 10.1.100.2:1194 -A nova-network-PREROUTING -d 10.1.0.76/32 -p udp -m udp --dport 1000 -j DNAT --to-destination 10.1.101.2:1194 -A nova-network-PREROUTING -d 10.1.0.89/32 -j DNAT --to-destination 10.1.100.8 -A nova-network-PREROUTING -d 10.1.0.84/32 -j DNAT --to-destination 10.1.100.9 -A nova-network-PREROUTING -d 10.1.0.85/32 -j DNAT --to-destination 10.1.100.10 -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.1.0.76:8773 -A nova-network-floating-snat -s 10.1.100.8/32 -j SNAT --to-source 10.1.0.89 -A nova-network-floating-snat -s 10.1.100.9/32 -j SNAT --to-source 10.1.0.84 -A nova-network-floating-snat -s 10.1.100.10/32 -j SNAT --to-source 10.1.0.85 -A nova-network-snat -j nova-network-floating-snat -A nova-network-snat -s 10.1.100.0/22 -j SNAT --to-source 10.1.0.76 -A nova-postrouting-bottom -j nova-network-snat COMMIT

Completed on Wed Aug 24 13:03:30 2011

Generated by iptables-save v1.4.4 on Wed Aug 24 13:03:30 2011

*filter :INPUT ACCEPT [1983975:182109122] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1432744:140808755] :nova-filter-top - [0:0] :nova-network-FORWARD - [0:0] :nova-network-INPUT - [0:0] :nova-network-OUTPUT - [0:0] :nova-network-local - [0:0] -A INPUT -j nova-network-INPUT -A FORWARD -j nova-filter-top -A FORWARD -j nova-network-FORWARD -A OUTPUT -j nova-filter-top -A OUTPUT -j nova-network-OUTPUT -A nova-filter-top -j nova-network-local -A nova-network-FORWARD -i br_vlan100 -j ACCEPT -A nova-network-FORWARD -o br_vlan100 -j ACCEPT -A nova-network-FORWARD -d 10.1.100.2/32 -p udp -m udp --dport 1194 -j ACCEPT -A nova-network-FORWARD -i br_vlan101 -j ACCEPT -A nova-network-FORWARD -o br_vlan101 -j ACCEPT -A nova-network-FORWARD -d 10.1.101.2/32 -p udp -m udp --dport 1194 -j ACCEPT COMMIT

edit flag offensive delete link more
0

answered 2011-08-24 17:51:51 -0500

It should be noted that I removed the dmz_cidr option from the nova.conf as this was working previously without it. With that enabled, I could ssh to the fixed_ip of an instance, however after about 45 seconds, the connection would hang.

The IP I've been using to test in this scenario is 10.1.100.11.

edit flag offensive delete link more
0

answered 2011-08-29 15:12:38 -0500

Hey there,

This was solved and due to a quirk in our network rather than a quirk with nova-network. We trunk our vlans from the border router down to the switches, and in doing so essentially provided multiple gateways for traffic to/from an instance. Removing the vlans from the BR so they exist only on the switchgear connected to the OpenStack instance have fixed this problem.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-08-24 17:22:57 -0500

Seen: 36 times

Last updated: Aug 29 '11