Ask Your Question
0

Also Nova+Quantum+Openvswitch VLAN Problem!

asked 2012-06-19 08:40:10 -0500

guestly gravatar image

My environment includes two physical hosts. One of them (domain name cc201) installed all of nova components and Glance, Quantum, Keystone, Horizon, Open-vSwitch as controller and network node; the other installed only nova-compute, Quantum, Open-vSwitch as compute node.

also I run quantum-agent, load 8021q module.etc Everything seems runs well. But I found a curious problem!

On cc201, I create networks 192.168.153.0/24(network3, its vlan ID is 6)、192.168.155.0/24(network5, its vlan ID is 8) I run nova-manage on host cc201 to create VMs. the VMs are: 192.168.153.2 (on host cc201) 192.168.153.4 (on host cc202) 192.168.153.5 (on host cc202) 192.168.153.6 (on host cc202) 192.168.155.2 (on host cc201) 192.168.155.3 (on host cc201) 192.168.155.4 (on host cc202)

I log on one of them to ping another of them and capture packets through eth1(eth1 interface the openvswitch port on both of my hosts cc201 and cc202, by using command ovs-vsctl add-port eth1 br-int), results are as below:

I run tcpdump -i eth1 -v -w to capture and save packets when I log on 192.168.153.2 (on cc201, in vlan6): ping 192.168.153.4 (on cc202, in vlan6): (result is they connected) on eth1 of cc201:I can see vlan 6 tag in ICMP request and reply, it is what I expected on eth1 of cc202:I can see vlan 6 tag in ICMP request and reply, it is what I expected

when I log on 192.168.153.2 (on cc201, in vlan 6): ping 192.168.155.2 (on cc201, in vlan8): (result is they are not connected) this result is also what I expect

But when I log on 192.168.153.2 (on cc201, in vlan 6) ping 192.168.155.4 (on cc202, in vlan 8): (result is they connected!!!) on eth1 of cc201: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!! on eth1 of cc202: I cannot see vlan 6 tag in ICMP, instead, I can see vlan 8 tag in ICMP request and reply!!! another words, it has the wrong vlan tag! (expect 6 but actually 8!)

so what happened ? Is this a known bug of quantum or openvswitch?

more details is as below: 1)nova config on cc201: nova.conf--http://paste.openstack.org/show/18588/ nova-compute.conf--http://paste.openstack.org/show/18589/ 2)nova config on cc202: nova.conf--http://paste.openstack.org/show/18590/ nova-compute.conf--http://paste.openstack.org/show/18591/ 3)other command results on cc201--http://paste.openstack.org/show/18592/ 4)other command results on cc202--http://paste.openstack.org/show/18593/

edit retag flag offensive close merge delete

28 answers

Sort by » oldest newest most voted
0

answered 2012-07-06 01:06:02 -0500

guestly gravatar image

Thanks mizumoto, that solved my question.

edit flag offensive delete link more
0

answered 2012-06-19 09:25:59 -0500

guestly gravatar image

output of the second capture of packets (ping from 192.168.153.2 in vlan 6 on cc201 to 192.168.155.4 in vlan 8 on cc202):

in_port(7),eth(src=fa:16:3e:1a:95:5b,dst=fa:16:3e:7f:6e:64),eth_type(0x0800),ipv4(src=192.168.153.2,dst=192.168.155.4,proto=1,tos=0,ttl=64,frag=no),icmp(type=8,code=0), packets:20, bytes:1960, used:0.192s, actions:6 in_port(1),eth(src=24:b6:57:f3:56:0b,dst=01:00:0c:cc:cc:cd),eth_type(0x8100),vlan(vid=26,pcp=7),encap(), packets:4549, bytes:291136, used:1.396s, actions:0 in_port(1),eth(src=54:52:48:00:08:13,dst=33:33:ff:00:08:13),eth_type(0x86dd),ipv6(src=fe80::5652:48ff:fe00:813,dst=ff02::1:ff00:813,label=0,proto=58,tclass=0,hlimit=1,frag=no),icmpv6(type=131,code=0), packets:0, bytes:0, used:never, actions:0 in_port(19),eth(src=fa:16:3e:22:d7:40,dst=fa:16:3e:1b:cf:e2),eth_type(0x0800),ipv4(src=192.168.153.2,dst=192.168.155.4,proto=1,tos=0,ttl=63,frag=no),icmp(type=8,code=0), packets:20, bytes:1960, used:0.192s, actions:push_vlan(vid=8,pcp=0),1

It says that operation push_vlan put the packets with tag 8 not tag 6!! I suppose it is the reason why two VMs from different vlan can ping connected!

But why???

edit flag offensive delete link more
0

answered 2012-06-20 07:32:32 -0500

Hi,

It seems to be connected through the default gateway of VM:192.168.153.2, that is, gw-xxx of host cc201 interface. It should be 192.168.153.1 and this gw-xxx interface forward to the other gw-yyy of host cc201 interface internally. it's address should be 192.168.155.1 for vlan ID 8. Then, br-int of cc201 send out from eth1 of cc201 to eth1 of cc202 with vlan ID 8 because its broadcast domain were 192.168.155.x. The br-int of cc201 just forward to tap device of VM:192.168.155.4.

I think it was normal behaviour. Please check each VM's routing table, and cc201/cc202 ifconfig, and the output of ovs-vsctl show.

mizumoto

edit flag offensive delete link more
0

answered 2012-06-21 04:12:43 -0500

The tag was attached / replaced when go out from cc201's eth1 to outside, in this case to cc202's eth1. The icmp is routed from vm of cc201 through 2 gw- interface, and finally decided to go outside. So last gw is 192.168.155.1 L2 domain, so the tag you watched on cc202 eth1 was tag: 8 because routed. That's one of reason the icmp has reached to cc202's VM even separate network.

However, it should be same behaviour when 1 vm send to another vm which is on other network even it was on other hypervisor or same hypervisor. It should be control by nova's security group or project separation of nova network setting. So in this viewpoint, nova configuration, iptables, or some other thing was wrong I think.

mizumoto

edit flag offensive delete link more
0

answered 2012-06-20 08:34:37 -0500

guestly gravatar image

Thanks to mizumoto for your reply? But why I ping from 192.168.153.2(on cc201, vlan 6) to 192.168.155.2(on cc201, vlan 8), it is not connected? Is it because they are on the same physical host? If so, how can I make VLAN work? What should I do?

Because I the image I use is simple tty-linux, so the information may be not detail. If you need more informations, just tell me! Routing tables of VMs are as below: 1) on 192.168.153.2: $ route
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.153.0 * 255.255.255.0 U 0 0 0 eth0 default 192.168.153.1 0.0.0.0 UG 0 0 0 eth0

2)on 192.168.155.4: $ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.155.0 * 255.255.255.0 U 0 0 0 eth0 default 192.168.155.1 0.0.0.0 UG 0 0 0 eth0

3) cc201 ovs-vsctl:

ovs-vsctl show

974585a6-2b5b-4bea-be4a-3127f71007bf Bridge vlan-test-br Port vlan-test-br Interface vlan-test-br type: internal Bridge br-int Port "gw-87a6f352-35" tag: 4 Interface "gw-87a6f352-35" type: internal Port "gw-e7863e30-1c" tag: 6 Interface "gw-e7863e30-1c" type: internal Port "tap908304d1-d1" tag: 8 Interface "tap908304d1-d1" Port "gw-6d53af57-d9" tag: 8 Interface "gw-6d53af57-d9" type: internal Port "tap1585a714-10" tag: 6 Interface "tap1585a714-10" Port br-int Interface br-int type: internal Port "eth1" Interface "eth1" Port "gw-3faffe59-5d" tag: 5 Interface "gw-3faffe59-5d" type: internal Port "tap09ae4556-2b" tag: 8 Interface "tap09ae4556-2b" Port "tap77918099-5a" tag: 6 Interface "tap77918099-5a" Port "tap6011d326-33" tag: 6 Interface "tap6011d326-33" Port "gw-9a2757f4-2d" tag: 7 Interface "gw-9a2757f4-2d" type: internal ovs_version: "1.4.0+build0"

4) cc201 ifconfig:

ifconfig

br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10524930 errors:0 dropped:0 overruns:0 frame:0 TX packets:9110 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:714364025 (714.3 MB) TX bytes:783412 (783.4 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:d4
inet addr:10.131.0.31 Bcast:10.131.255.255 Mask:255.252.0.0 inet6 addr: fe80::d267:e5ff:fe23:96d4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3473405 errors:0 dropped:0 overruns:0 frame:0 TX packets:2372097 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:694315585 (694.3 MB) TX bytes:1224320432 (1.2 GB) Interrupt:43 Base address:0x6000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:10906571 errors:0 dropped:0 overruns:0 frame:0 TX packets:99047 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:814630415 (814.6 MB) TX ...
(more)

edit flag offensive delete link more
0

answered 2012-06-20 08:34:40 -0500

guestly gravatar image

Thanks to mizumoto for your reply? But why I ping from 192.168.153.2(on cc201, vlan 6) to 192.168.155.2(on cc201, vlan 8), it is not connected? Is it because they are on the same physical host? If so, how can I make VLAN work? What should I do?

Because I the image I use is simple tty-linux, so the information may be not detail. If you need more informations, just tell me! Routing tables of VMs are as below: 1) on 192.168.153.2: $ route
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.153.0 * 255.255.255.0 U 0 0 0 eth0 default 192.168.153.1 0.0.0.0 UG 0 0 0 eth0

2)on 192.168.155.4: $ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.155.0 * 255.255.255.0 U 0 0 0 eth0 default 192.168.155.1 0.0.0.0 UG 0 0 0 eth0

3) cc201 ovs-vsctl:

ovs-vsctl show

974585a6-2b5b-4bea-be4a-3127f71007bf Bridge vlan-test-br Port vlan-test-br Interface vlan-test-br type: internal Bridge br-int Port "gw-87a6f352-35" tag: 4 Interface "gw-87a6f352-35" type: internal Port "gw-e7863e30-1c" tag: 6 Interface "gw-e7863e30-1c" type: internal Port "tap908304d1-d1" tag: 8 Interface "tap908304d1-d1" Port "gw-6d53af57-d9" tag: 8 Interface "gw-6d53af57-d9" type: internal Port "tap1585a714-10" tag: 6 Interface "tap1585a714-10" Port br-int Interface br-int type: internal Port "eth1" Interface "eth1" Port "gw-3faffe59-5d" tag: 5 Interface "gw-3faffe59-5d" type: internal Port "tap09ae4556-2b" tag: 8 Interface "tap09ae4556-2b" Port "tap77918099-5a" tag: 6 Interface "tap77918099-5a" Port "tap6011d326-33" tag: 6 Interface "tap6011d326-33" Port "gw-9a2757f4-2d" tag: 7 Interface "gw-9a2757f4-2d" type: internal ovs_version: "1.4.0+build0"

4) cc201 ifconfig:

ifconfig

br-int Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10524930 errors:0 dropped:0 overruns:0 frame:0 TX packets:9110 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:714364025 (714.3 MB) TX bytes:783412 (783.4 KB)

eth0 Link encap:Ethernet HWaddr d0:67:e5:23:96:d4
inet addr:10.131.0.31 Bcast:10.131.255.255 Mask:255.252.0.0 inet6 addr: fe80::d267:e5ff:fe23:96d4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3473405 errors:0 dropped:0 overruns:0 frame:0 TX packets:2372097 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:694315585 (694.3 MB) TX bytes:1224320432 (1.2 GB) Interrupt:43 Base address:0x6000

eth1 Link encap:Ethernet HWaddr 00:1b:21:c6:35:a6
inet6 addr: fe80::21b:21ff:fec6:35a6/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:10906571 errors:0 dropped:0 overruns:0 frame:0 TX packets:99047 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:814630415 (814.6 MB) TX ...
(more)

edit flag offensive delete link more
0

answered 2012-06-20 09:27:08 -0500

Let me clear your information. You mentioned 3VMs and just 1 VM on VLAN tag: 6, i.e. 192.168.153.2, on cc201 but,

I run nova-manage on host cc201 to create VMs. the VMs are: 192.168.153.2 (on host cc201) 192.168.153.4 (on host cc202) 192.168.153.5 (on host cc202) 192.168.153.6 (on host cc202) 192.168.155.2 (on host cc201) 192.168.155.3 (on host cc201) 192.168.155.4 (on host cc202)

the cc201 OVS have 3 tap device and 1 gw on VLAN tag: 6:

    Port "tap6011d326-33"
        tag: 6
    Port "gw-e7863e30-1c"
        tag: 6
    Port "tap1585a714-10"
        tag: 6
    Port "tap77918099-5a"

tag: 6

And 2 VM on cc201 with VLAN tag: 8, but this don't have difference: Port "tap09ae4556-2b" tag: 8 Port "tap908304d1-d1" tag: 8 Port "gw-6d53af57-d9"

tag: 8

Did you just add more 2 VM on VLAN tag 6 on OVS of cc201, so have 2 more tap on VLAN tag 6, even not mention before?

mizumoto

edit flag offensive delete link more
0

answered 2012-06-20 09:31:01 -0500

guestly gravatar image

Yes, after that, I created moure VM. But I think it is not concern with the problem. Do you think so?

edit flag offensive delete link more
0

answered 2012-06-20 09:34:24 -0500

guestly gravatar image

Network 192.168.153.0/24 has vlan6; Network 192.168.155.0/24 has vlan8.

edit flag offensive delete link more
0

answered 2012-06-20 10:45:33 -0500

I created similar environment with 2 compute node with OVS quantum. Among 2 nova-network communication is controlled by iptables. And your output don't have ICMP for each instance. So expected behaviour, each VM just communicate with same network. I changed security group setting to pass ICMP, each VM could communicate with other network.

However, I could duplicated your behaviour. So something wrong and have to investigate more.

http://paste.openstack.org/show/18639/

Yes, adding more VM with correct tap, there is no problem. I asked why I had some problem with old tap/gw device had odd behaviour before.

mizumoto

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-06-19 08:40:10 -0500

Seen: 209 times

Last updated: Jul 18 '12