Ask Your Question
0

Also Nova+Quantum+Openvswitch VLAN Problem!

asked 2012-06-19 08:40:10 -0500

guestly gravatar image

My environment includes two physical hosts. One of them (domain name cc201) installed all of nova components and Glance, Quantum, Keystone, Horizon, Open-vSwitch as controller and network node; the other installed only nova-compute, Quantum, Open-vSwitch as compute node.

also I run quantum-agent, load 8021q module.etc Everything seems runs well. But I found a curious problem!

On cc201, I create networks 192.168.153.0/24(network3, its vlan ID is 6)、192.168.155.0/24(network5, its vlan ID is 8) I run nova-manage on host cc201 to create VMs. the VMs are: 192.168.153.2 (on host cc201) 192.168.153.4 (on host cc202) 192.168.153.5 (on host cc202) 192.168.153.6 (on host cc202) 192.168.155.2 (on host cc201) 192.168.155.3 (on host cc201) 192.168.155.4 (on host cc202)

I log on one of them to ping another of them and capture packets through eth1(eth1 interface the openvswitch port on both of my hosts cc201 and cc202, by using command ovs-vsctl add-port eth1 br-int), results are as below:

I run tcpdump -i eth1 -v -w to capture and save packets when I log on 192.168.153.2 (on cc201, in vlan6): ping 192.168.153.4 (on cc202, in vlan6): (result is they connected) on eth1 of cc201:I can see vlan 6 tag in ICMP request and reply, it is what I expected on eth1 of cc202:I can see vlan 6 tag in ICMP request and reply, it is what I expected

when I log on 192.168.153.2 (on cc201, in vlan 6): ping 192.168.155.2 (on cc201, in vlan8): (result is they are not connected) this result is also what I expect

But when I log on 192.168.153.2 (on cc201, in vlan 6) ping 192.168.155.4 (on cc202, in vlan 8): (result is they connected!!!) on eth1 of cc201: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!! on eth1 of cc202: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!! another words, it has the wrong vlan tag! (expect 6 but actually 8!)

so what happened ? Is this a known bug of quantum or openvswitch?

more details is as below: 1)nova config on cc201: nova.conf--http://paste.openstack.org/show/18588/ nova-compute.conf--http://paste.openstack.org/show/18589/ 2)nova config on cc202: nova.conf--http://paste.openstack.org/show/18590/ nova-compute.conf--http://paste.openstack.org/show/18591/ 3)other command results on cc201--http://paste.openstack.org/show/18592/ 4)other command results on cc202--http://paste.openstack.org/show/18593/

edit retag flag offensive close merge delete

28 answers

Sort by » oldest newest most voted
0

answered 2012-06-20 12:34:28 -0500

guestly gravatar image

mizumoto: Thanks for your reply! From your paste, you created two networks (two vlans), one is 172.15.3.0, the other is 172.15.5.0. If vlan does works, these two networks should not be connected! In other words, if you log on 172.15.5.3 to ping 172.15.5.5, it is connected-----It is right! If you log on 172.15.5.3 to ping 172.15.3.3 or 172.15.3.2, both of the two result should be not connected! But your test result is similar with mine. I don't know why~~

edit flag offensive delete link more
0

answered 2012-06-20 12:37:56 -0500

guestly gravatar image

And, "Among 2 nova-network communication is controlled by iptables. " May I ask how do you set your iptables configuration? And, "I changed security group setting to pass ICMP, each VM could communicate with other network." What does your security group setting look like?

edit flag offensive delete link more
0

answered 2012-06-20 15:43:40 -0500

My understanding is tap device and OVS vlan tag works correct, however, iptables or forwarding between gw interfaces for outside interface not work. I see the tcpdump of routed gw interface, source IP address has changed (NAT or routing_source_ip of nova.conf?) not nova network but KVM's physical IP address.

localadmin@kvm155:~$ sudo tcpdump -i gw-0abd2a02-ed -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on gw-0abd2a02-ed, link-type EN10MB (Ethernet), capture size 65535 bytes 00:41:20.463956 IP 172.15.5.3 > 172.15.3.3: ICMP echo request, id 48129, seq 0, length 64 00:41:20.464752 IP 172.15.3.3 > 172.15.5.3: ICMP echo reply, id 48129, seq 0, length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel localadmin@kvm155:~$ sudo tcpdump -i gw-a9689dbb-dd -n icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on gw-a9689dbb-dd, link-type EN10MB (Ethernet), capture size 65535 bytes 00:41:46.571095 IP 10.127.1.155 > 172.15.3.3: ICMP echo request, id 48385, seq 0, length 64 00:41:46.571866 IP 172.15.3.3 > 10.127.1.155: ICMP echo reply, id 48385, seq 0, length 64

^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel localadmin@kvm155:~$ sudo tcpdump -i eth1 -n icmp tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 00:42:26.610185 IP 10.127.1.155 > 172.15.3.3: ICMP echo request, id 50689, seq 0, length 64 00:42:26.610858 IP 172.15.3.3 > 10.127.1.155: ICMP echo reply, id 50689, seq 0, length 64

To change security group setting, just add ICMP rules for applied security group of the VM. In my case, default security group.

http://paste.openstack.org/show/18649/

mizumoto

edit flag offensive delete link more
0

answered 2012-06-20 16:33:30 -0500

I checked adding iptable entry manually, it could block the packet among networks. However, this is not compatible security group and drop all communication among networks, so this is just information and testing purpose but this is the rule:

[Append rule] iptables -t filter -A nova-network-FORWARD --in-interface gw-+ --out-interface gw-+ -j DROP

[Delete rule] iptables -t filter -D nova-network-FORWARD --in-interface gw-+ --out-interface gw-+ -j DROP

mizumoto

edit flag offensive delete link more
0

answered 2012-06-21 01:16:19 -0500

guestly gravatar image

"I checked adding iptable entry manually, it could block the packet among networks." Hmmm, I understand what you do. But should it be quantum's work? If we must add iptable entry to solve this problem manually, it will be a few entries when there are many networks! So I think there must somthing wrong with Quantum or Openvswitch. But I've never seen otherone reporting them:-(

But why I ping from networkA(on cc201) to networkB(also on cc201), the vlan can block the ping packets?

this morning, I checked the command iptables-save, and found these rules, do they have some relation with this problem?

-A nova-compute-local -d 192.168.153.2/32 -j nova-compute-inst-6 -A nova-compute-local -d 192.168.155.2/32 -j nova-compute-inst-22 -A nova-compute-local -d 192.168.155.3/32 -j nova-compute-inst-23 -A nova-compute-local -d 192.168.153.8/32 -j nova-compute-inst-26 -A nova-compute-local -d 192.168.153.9/32 -j nova-compute-inst-27 -A nova-compute-sg-fallback -j DROP -A nova-filter-top -j nova-compute-local -A nova-filter-top -j nova-network-local -A nova-filter-top -j nova-manage-local -A nova-filter-top -j nova-api-local -A nova-network-FORWARD -i br-int -j ACCEPT -A nova-network-FORWARD -o br-int -j ACCEPT -A nova-network-INPUT -i gw-e7863e30-1c -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-e7863e30-1c -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-e7863e30-1c -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-e7863e30-1c -p tcp -m tcp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-87a6f352-35 -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-87a6f352-35 -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-87a6f352-35 -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-87a6f352-35 -p tcp -m tcp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-3faffe59-5d -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-3faffe59-5d -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-3faffe59-5d -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-3faffe59-5d -p tcp -m tcp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-6d53af57-d9 -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-6d53af57-d9 -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-6d53af57-d9 -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-6d53af57-d9 -p tcp -m tcp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-9a2757f4-2d -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-9a2757f4-2d -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i gw-9a2757f4-2d -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i gw-9a2757f4-2d -p tcp -m tcp --dport 53 -j ACCEPT

edit flag offensive delete link more
0

answered 2012-06-21 01:26:01 -0500

guestly gravatar image

to mizumoto: "source IP address has changed (NAT or routing_source_ip of nova.conf?) "

I have something similar with you before. I think this because the nova-network add a rule like: -A nova-network-snat -s 192.168.151.0/24 -j SNAT --to-source 10.131.0.31

I delete this rule. And thus, the source address won't be changed.

This is one of the problems, but I think it is not critical.

The critical problem also is why the vlan tag wrong or another words why it is connected between different gw*s?

edit flag offensive delete link more
0

answered 2012-06-21 02:19:53 -0500

Havent,

The rule I wrote for in/out gw interface is just adding to rest of normal iptables entry, the most of them generated for each instances. The original iptables are here:

http://paste.openstack.org/show/18675/ http://paste.openstack.org/show/18676/

How to know quantum work or not is to see ovs-vsctl show for tap device and agent VLAN setting for br-int with ovs_quantum db entry. Run quantum-agent with debug option.

I don't think vlan tag was wrong.

In your case:

cc201:192.168.153.2(tap)--->br-int-->(gw)192.168.153.1--->(gw)192.168.155.1--->br-int--->cc201:eth1---> ---->[tag:8]--sw--[tag:8]---> --->cc202:eth1--->br-int--->(tap)cc202:192.168.155.2

So you watched cc201:eth1, the VLAN was tag:8. I inserted iptables rule between in/out gateway for testing purpose in my environment.

mizumoto

edit flag offensive delete link more
0

answered 2012-06-21 03:58:03 -0500

guestly gravatar image

Yes, in my case, the packets' path is just like: cc201:192.168.153.2(tap)--->br-int-->(gw)192.168.153.1--->(gw)192.168.155.1--->br-int--->cc201:eth1---> ---->[tag:8]--sw--[tag:8]---> --->cc202:eth1--->br-int--->(tap)cc202:192.168.155.2

But, I mean 192.168.153.2 is under network 192.168.153.0/24, it is vlan6 not vlan8, why it has tag8(vlan8's tag) finally? I mean if openvswitch know icmp's source is a vlan6 VM, and its destination is a vlan8 VM, the ovs should prevent the vlan6 packets to access and drop it. Do I have the correct understanding?

edit flag offensive delete link more
0

answered 2012-06-21 04:42:58 -0500

guestly gravatar image

"So in this viewpoint, nova configuration, iptables, or some other thing was wrong I think." I agree with this point of view. on the same hypervisor, the result is not connected, but on the different hypervisor, it is connected. I had reported this problem in bugs series, but I don't know who will tell us the result.

I will reinstall my environment and have a try. If this time the problem is not resolved, I think waiting for someone to analyze and resolve it is the best choice for us.

edit flag offensive delete link more
0

answered 2012-06-21 05:36:03 -0500

In my environment, the below routing_source_ip setting of nova.conf disabled the floating nat of iptables (nova-network-float-snat). This is you mentioned above.


nova.conf

--routing_source_ip=0.0.0.0

With changing this value, and restart nova-network will change snat of iptables. http://paste.openstack.org/show/18677/

Then, the network had separated in my environment. That is, the vm can communicate with other vm only in same network. So floating IP setting will change behaviour. I'm not sure which setting will be reasonable in this kind of environment, but I think you don't need reinstall for this.

mizumoto

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-06-19 08:40:10 -0500

Seen: 209 times

Last updated: Jul 18 '12