Ask Your Question
0

Quantum OVS multiple networks

asked 2012-03-16 15:43:03 -0500

mb-s gravatar image

Here is a question that I have not been able to find an answer to. I try to make the problem as clear as possible:

I have two servers: Server A and Server B. Server A installed using devstack runs all the services including Quantum with OVS. Server B is a Compute node only with Quantum OVS agent.

I create two networks: one for the Admin project (9.9.9.0/24) and one for the Demo project (9.9.8.0/24) .

Then I start creating VMs for each project which get created alternatively on Server A and Server B. After creating a few VMs I have the following:

Server A VMs: admin1, admin3, admin5, demo1, demo3, demo5 (all having correct IP like 9.9.9.2 and 9.9.8.2, etc) Server B VMs: admin2, admin4, admin6, demo2, demo4, demo6 (all having correct IP like 9.9.9.3 and 9.9.8.3, etc)

Now from VMs on each network I can ping other VMs on the same network. So far so good.

Then I try pining nodes on the other network and I notice the following:

From each VM on Server A I can also ping the VMs on Server B which are on the other network. For example from admin1, I can ping demo2, demo4, and demo6 even though I cannot ping demo1, demo3, and demo5.

Why? What am I doing wrong?

This is the traceroute from 9.9.9.2 on Server A to 9.9.8.4 on server B. The ping works while it should not.

% traceroute to 9.9.8.4 (9.9.8.4), 30 hops max, 46 byte packets 1 http://reserved-9-9-9-1.atlanta.ibm.com (9.9.9.1) 0.477 ms 0.183 ms 0.188 ms 2 9.9.8.4 (9.9.8.4) 0.988 ms 0.596 ms 0.522 ms

This is the traceroute from 9.9.9.2 on Server A to 9.9.8.5 on Server A. The ping does not work as expected: % traceroute to 9.9.8.5 (9.9.8.5), 30 hops max, 46 byte packets 1 http://reserved-9-9-9-1.atlanta.ibm.com (9.9.9.1) 0.444 ms 0.239 ms 0.223 ms 2 * * * 3 * * *

Here is the route table on Server A:

mb@sysnet45:~$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 9.2.156.65 0.0.0.0 UG 0 0 0 eth1 9.2.156.64 * 255.255.255.192 U 0 0 0 eth1 9.9.8.0 * 255.255.255.0 U 0 0 0 gw-f25ffc5e-d3 9.9.9.0 * 255.255.255.0 U 0 0 0 gw-e84cecf1-06 10.0.0.0 * 255.255.255.0 U 0 0 0 gw-9ad6270b-f7 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0 192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0

Here is the ... (more)

edit retag flag offensive close merge delete

18 answers

Sort by ยป oldest newest most voted
0

answered 2012-03-20 17:40:44 -0500

bgh gravatar image

Hi Mohammad,

Can you also paste the output of iptables -L -n -t nat?

edit flag offensive delete link more
0

answered 2012-03-20 22:17:53 -0500

mb-s gravatar image

Sure.

On Server A:

mb@sysnet45:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=411937.037s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=3 actions=drop cookie=0x0, duration=411934.967s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=2 actions=drop cookie=0x0, duration=411334.842s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=7 actions=drop cookie=0x0, duration=412155.689s, table=0, n_packets=882468, n_bytes=111928515, priority=1 actions=NORMAL

============================================================================

mb@sysnet43:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=411739.341s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=1 actions=drop cookie=0x0, duration=370942.723s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=5 actions=drop cookie=0x0, duration=412054.101s, table=0, n_packets=582126, n_bytes=71555265,priority=1 actions=NORMAL mb@sysnet43:~$

edit flag offensive delete link more
0

answered 2012-03-23 19:20:22 -0500

mb-s gravatar image

Just wondering if you found any clues as to what may be going on here? Thanks.

edit flag offensive delete link more
0

answered 2012-03-17 22:35:23 -0500

salvatore-orlando gravatar image

Hi Mohammad,

very interesting question. Thanks for posting such a great deal of details!

Assuming "admin" and "demo" are your "tenants", it looks like "admin" VM on server A cannot ping "demo" VMs on server A while they can ping them on server B, while they should not.

The traceroute seems to show a packet leaving "admin" VM on server A, going through some router/gateway, and finally reaching a VM for tenant "demo" on server B. If that is correct, the traffic is leaving the Quantum network on server A, and then is getting back in it on server B; in this case I wonder how OVS has been configured on the two hosts and whether IP routing is changing the way in which the packets are being forwarded between the two hosts.

From the routing table for server A, I some routes specific for the tenant networks (9.9.8.0/24 and 9.9.9.0/24).
1) Are they routing the packets for these networks to a router which performs VLAN termination? 2) How are the OVS instances on server A and server B connected each other? 3) Are there interfaces different from VIFs, GRE tunnels, and 'patch interfaces' plugged into the OVS instances?

Regards, Salvatore

edit flag offensive delete link more
0

answered 2012-03-19 14:27:50 -0500

mb-s gravatar image

Thanks for the response. Server A and Server B are connected to each other through their eth1 interfaces which are connected to a single switch with no other connections. (eth0 on both servers are connected to our internal network and Internet.) I am ot sure about what this particular switch does but I would thing it is not tha cause of our issues because traffic gets passed through it for some of the ping operations.

There are no other interfaces of the OVS as listed below.

Please let me know if there are other pieces of information that can be helpful.

Thanks,

-Mohammad

On Server A: The three gateways are for my three networks and the tap interfaces are for the four running VMs right now.

mb@sysnet45:~$ sudo ovs-vsctl list-br br-int mb@sysnet45:~$ sudo ovs-vsctl list-ports br-int eth0 gw-9ad6270b-f7 gw-e84cecf1-06 gw-f25ffc5e-d3 tapaf08a421-b7 tapc9e6c971-b4 tapd56b38e0-66 tapf059b70a-ec

On Server B: the tap interfaces are for the five running VMs right now.

mb@sysnet43:~$ sudo ovs-vsctl list-br br-int mb@sysnet43:~$ sudo ovs-vsctl list-ports br-int eth0 tap062e466f-c7 tap0abbbcb6-72 tap2d7c1378-fe tap92d4d472-95 tap961b958b-d1

edit flag offensive delete link more
0

answered 2012-03-19 16:11:08 -0500

danwent gravatar image

Hi Mohammad, thanks for the detailed write-up. Seems like this may be a bug.

I'd like to clarify one thing though: the "correct" behavior is actually that VMs from the two networks CAN reach each other, but only after traversing an L3 hop. This at least, is based on a discussion I had with Vish about how VLANManager works (which is essentially what Quantum emulates if you create per-project networks). I think the reason it works that way is that they were emulating Amazon, where you have "internal" IPs that can all reach each other, then public floating IPs that cannot. In Folsom Quantum will get rid of the old nova networking L3 code and will support much richer configuration of L3 topologies. In the mean time, we're stuck with what was in Nova.

So with that in mind, the real question is why we can't connect to VMs in the other subnet when they are on the same host. Running tcpdump on the gateway interfaces that should be receiving and forwarding the traffic should be informative. The devices are named with the pattern gw-*, where * is the start of the network uuid visible if you run "quantum list_nets <tenant-id>" or "nova-manage network quantum_list". In the case where traffic does not flow, it would be interesting to see whether the traffic is reaching the gateway device for the 9.9.9.0/24 subnet, and if so, whether it is leaving the gateway device for the 9.9.8.0/24 subnet.

edit flag offensive delete link more
0

answered 2012-03-19 17:53:32 -0500

mb-s gravatar image

Looks like things do not get passed the 9.9.9.0 gateway.

Below I am copying the output for both gateways and for two cases: first where the ping is not successful and then the case where ping is successful. (I can attach the complete output of these files if i can figure out how to do it here.)

Case 1 - Here is the case where 9.9.9.2 on Server A canNOT ping 9.9.8.5 on Server A:

on the 9.9.9.0 gatway:

13:44:29.433031 IP (tos 0xc0, ttl 64, id 7316, offset 0, flags [none], proto ICMP (1), length 367) 9.9.9.2 > 9.2.156.126: ICMP 9.9.9.2 udp port 68 unreachable, length 347 IP (tos 0x0, ttl 64, id 58981, offset 0, flags [none], proto UDP (17), length 339) 9.2.156.126.67 > 9.9.9.2.68: [udp sum ok] BOOTP/DHCP, Reply, length 311, xid 0x7a24b831, Flags [none] (0x0000) Client-IP 9.9.9.2 Your-IP 9.9.9.2 Server-IP 9.9.9.1 Client-Ethernet-Address 02:16:3e:22:06:15 Vendor-rfc1048 Extensions Magic Cookie 0x63825363 DHCP-Message Option 53, length 1: ACK Server-ID Option 54, length 4: 9.9.9.1 Lease-Time Option 51, length 4: 120 RN Option 58, length 4: 56 RB Option 59, length 4: 101 Subnet-Mask Option 1, length 4: 255.255.255.0 BR Option 28, length 4: 9.9.9.255 Default-Gateway Option 3, length 4: 9.9.9.1 Domain-Name-Server Option 6, length 4: 9.9.9.1 Domain-Name Option 15, length 9: "novalocal" Hostname Option 12, length 6: "host-9" 13:44:29.745432 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 9.9.9.2 > 9.9.8.5: ICMP echo request, id 26981, seq 13, length 64 13:44:30.746177 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 9.9.9.2 > 9.9.8.5: ICMP echo request, id 26981, seq 14, length 64 13:44:31.746438 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 9.9.9.2 > 9.9.8.5: ICMP echo request, id 26981, seq 15, length 64

on the 9.9.8.0 gatway:

nothing here

--------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------

Case 2 - Here is the case where 9.9.9.2 on Server A can ping 9.9.8.4 on Server B:

on the 9.9.9.0 gatway:

13:39:03.856916 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 9.9.9.2 > 9.9.8.4: ICMP echo request, id 17765, seq 96, length 64 13:39:03.857260 IP (tos 0x0, ttl 63, id 6921, offset 0, flags [none], proto ICMP (1), length 84) 9.9.8.4 > 9.9.9.2: ICMP echo reply, id 17765, seq 96, length 64 ... (more)

edit flag offensive delete link more
0

answered 2012-03-20 06:55:20 -0500

danwent gravatar image

Hi, I noticed that in the case that the ping works, the source IP address in the tcpdump changes by the time the traffic is exiting the 9.9.8.0 gateway. I suspect that the traffic is actually being SNATed. Is 9.2.156.126 the "public" IP of your network node?

If so, I think it would be useful to see the iptables rules on that host.

edit flag offensive delete link more
0

answered 2012-03-20 12:35:42 -0500

salvatore-orlando gravatar image

I agree with Dan that SNAT appears to be occuring when a packet is sent from server B. It also seems that SNAT is not occuring from server A to server B (I would have expected the 9.9.8.0 to receive pkt with altered src IP as the 9.9.9.0 gateway)

iptables rules would be extremely interesting. Understanding whether 9.2.156.126 belongs to the nova's network node (as public IP) or is just the IP address associated with eth1 could also be helpful.

You might provide also (if possible) - output from ovs-dpctl dump-flows br-int on both servers (I typically run it in a rather rude form: watch --interval=0.5 "date >> output.txt; ovs-dpctl dump-flows br-int >> output.txt"). This will let us understand where packets sents from server B are forwarded and vice-versa.

You can use http://paste.openstack.org for pasting large chunks of text.

Salvatore

edit flag offensive delete link more
0

answered 2012-03-20 16:14:21 -0500

mb-s gravatar image

Yes, 9.2.156.126 is the public IP of Server A (eth0). Here is the output of iptables for this server followed by that of Server B (9.2.156.124):

mb@sysnet45: ~mb@sysnet45:~$ iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination
nova-compute-INPUT all -- anywhere anywhere
nova-network-INPUT all -- anywhere anywhere
nova-manage-INPUT all -- anywhere anywhere
nova-api-INPUT all -- anywhere anywhere
ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT gre -- anywhere anywhere

Chain FORWARD (policy ACCEPT) target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-compute-FORWARD all -- anywhere anywhere
nova-network-FORWARD all -- anywhere anywhere
nova-manage-FORWARD all -- anywhere anywhere
nova-api-FORWARD all -- anywhere anywhere
ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT) target prot opt source destination
nova-filter-top all -- anywhere anywhere
nova-compute-OUTPUT all -- anywhere anywhere
nova-network-OUTPUT all -- anywhere anywhere
nova-manage-OUTPUT all -- anywhere anywhere
nova-api-OUTPUT all -- anywhere anywhere

Chain nova-api-FORWARD (1 references) target prot opt source destination

Chain nova-api-INPUT (1 references) target prot opt source destination
ACCEPT tcp -- anywhere http://sysnet45.watson.ibm.com tcp dpt:8775

Chain nova-api-OUTPUT (1 references) target prot opt source destination

Chain nova-api-local (1 references) target prot opt source destination

Chain nova-compute-FORWARD (1 references) target prot opt source destination

Chain nova-compute-INPUT (1 references) target prot opt source destination

Chain nova-compute-OUTPUT (1 references) target prot opt source destination

Chain nova-compute-inst-12 (1 references) target prot opt source destination
DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED nova-compute-provider all -- anywhere anywhere
ACCEPT udp -- http://reserved-9-9-9-1.atlanta.ibm.com anywhere udp spt:bootps dpt:bootpc ACCEPT all -- 9.9.9.0/24 anywhere
nova-compute-sg-fallback all -- anywhere anywhere

Chain nova-compute-inst-3 (1 references) target prot opt source destination
DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED nova-compute-provider all -- anywhere anywhere
ACCEPT udp -- sysnet45.local anywhere udp spt:bootps dpt:bootpc ACCEPT all -- 9.9.8.0/24 anywhere
nova-compute-sg-fallback all -- anywhere anywhere

Chain nova-compute-inst-7 (1 references) target prot opt source destination
DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED nova-compute-provider all -- anywhere anywhere
ACCEPT udp -- http://reserved-9-9-9-1.atlanta.ibm.com anywhere udp spt:bootps dpt:bootpc ACCEPT all -- 9.9.9.0/24 anywhere
nova-compute-sg-fallback all -- anywhere anywhere

Chain nova-compute-inst-9 (1 references) target prot opt source destination
DROP all -- anywhere anywhere state INVALID ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED nova-compute-provider all -- anywhere anywhere
ACCEPT udp -- sysnet45.local anywhere udp spt:bootps dpt:bootpc ACCEPT all -- 9.9.8.0/24 anywhere
nova-compute-sg-fallback all -- anywhere anywhere

Chain nova-compute-local (1 references) target prot opt source destination
nova-compute-inst-3 all -- anywhere 9.9.8.2
nova-compute-inst-7 all -- anywhere http://reserved-9-9-9-2.atlanta.ibm.com nova-compute-inst-9 all -- anywhere 9.9.8.5
nova-compute-inst-12 all -- anywhere http://reserved-9-9-9-4.atlanta.ibm.com

Chain ... (more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-03-16 15:43:03 -0500

Seen: 97 times

Last updated: May 17 '12