Ask Your Question
0

Quantum OVS multiple networks

asked 2012-03-16 15:43:03 -0500

mb-s gravatar image

Here is a question that I have not been able to find an answer to. I try to make the problem as clear as possible:

I have two servers: Server A and Server B. Server A installed using devstack runs all the services including Quantum with OVS. Server B is a Compute node only with Quantum OVS agent.

I create two networks: one for the Admin project (9.9.9.0/24) and one for the Demo project (9.9.8.0/24) .

Then I start creating VMs for each project which get created alternatively on Server A and Server B. After creating a few VMs I have the following:

Server A VMs: admin1, admin3, admin5, demo1, demo3, demo5 (all having correct IP like 9.9.9.2 and 9.9.8.2, etc) Server B VMs: admin2, admin4, admin6, demo2, demo4, demo6 (all having correct IP like 9.9.9.3 and 9.9.8.3, etc)

Now from VMs on each network I can ping other VMs on the same network. So far so good.

Then I try pining nodes on the other network and I notice the following:

From each VM on Server A I can also ping the VMs on Server B which are on the other network. For example from admin1, I can ping demo2, demo4, and demo6 even though I cannot ping demo1, demo3, and demo5.

Why? What am I doing wrong?

This is the traceroute from 9.9.9.2 on Server A to 9.9.8.4 on server B. The ping works while it should not.

% traceroute to 9.9.8.4 (9.9.8.4), 30 hops max, 46 byte packets 1 http://reserved-9-9-9-1.atlanta.ibm.com (9.9.9.1) 0.477 ms 0.183 ms 0.188 ms 2 9.9.8.4 (9.9.8.4) 0.988 ms 0.596 ms 0.522 ms

This is the traceroute from 9.9.9.2 on Server A to 9.9.8.5 on Server A. The ping does not work as expected: % traceroute to 9.9.8.5 (9.9.8.5), 30 hops max, 46 byte packets 1 http://reserved-9-9-9-1.atlanta.ibm.com (9.9.9.1) 0.444 ms 0.239 ms 0.223 ms 2 * * * 3 * * *

Here is the route table on Server A:

mb@sysnet45:~$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default 9.2.156.65 0.0.0.0 UG 0 0 0 eth1 9.2.156.64 * 255.255.255.192 U 0 0 0 eth1 9.9.8.0 * 255.255.255.0 U 0 0 0 gw-f25ffc5e-d3 9.9.9.0 * 255.255.255.0 U 0 0 0 gw-e84cecf1-06 10.0.0.0 * 255.255.255.0 U 0 0 0 gw-9ad6270b-f7 192.168.1.0 * 255.255.255.0 U 0 0 0 eth0 192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0

Here is the ... (more)

edit retag flag offensive close merge delete

18 answers

Sort by ยป oldest newest most voted
0

answered 2012-05-17 01:22:00 -0500

guestly gravatar image

I also have this problem. I founded that my iptables has a new rule after I created a network 192.168.208.0/24 and a VM 8.8.8.6/192.168.208.2, and I can ping 8.8.8.6 but cannot 192.168.208.2 :

-A nova-manage-snat -s 192.168.208.0/24 -j SNAT --to-source 10.131.0.244

when I delete this rule, I can ping 192.168.208.2, and when I create a new network and new VM, this rule was replaced by the new network.

As below, are my iptables rules BEFORE CREATING NETWORK, AFTER CREATING NETWORK and AFTER CREATING VM:

BEFORE CREATING NETWORK:

Generated by iptables-save v1.4.12 on Wed May 16 14:51:31 2012

*mangle :PREROUTING ACCEPT [245736:216294003] :INPUT ACCEPT [57864:32422084] :FORWARD ACCEPT [184175:182659303] :OUTPUT ACCEPT [53384:31506313] :POSTROUTING ACCEPT [237564:214167256] -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill COMMIT

Completed on Wed May 16 14:51:31 2012

Generated by iptables-save v1.4.12 on Wed May 16 14:51:31 2012

*nat :PREROUTING ACCEPT [135:35818] :INPUT ACCEPT [54:10054] :OUTPUT ACCEPT [22:1351] :POSTROUTING ACCEPT [22:1351] :nova-api-OUTPUT - [0:0] :nova-api-POSTROUTING - [0:0] :nova-api-PREROUTING - [0:0] :nova-api-float-snat - [0:0] :nova-api-snat - [0:0] :nova-compute-OUTPUT - [0:0] :nova-compute-POSTROUTING - [0:0] :nova-compute-PREROUTING - [0:0] :nova-compute-float-snat - [0:0] :nova-compute-snat - [0:0] :nova-manage-OUTPUT - [0:0] :nova-manage-POSTROUTING - [0:0] :nova-manage-PREROUTING - [0:0] :nova-manage-float-snat - [0:0] :nova-manage-snat - [0:0] :nova-network-OUTPUT - [0:0] :nova-network-POSTROUTING - [0:0] :nova-network-PREROUTING - [0:0] :nova-network-float-snat - [0:0] :nova-network-snat - [0:0] :nova-postrouting-bottom - [0:0] -A PREROUTING -j nova-compute-PREROUTING -A PREROUTING -j nova-network-PREROUTING -A PREROUTING -j nova-manage-PREROUTING -A PREROUTING -j nova-api-PREROUTING -A OUTPUT -j nova-compute-OUTPUT -A OUTPUT -j nova-network-OUTPUT -A OUTPUT -j nova-manage-OUTPUT -A OUTPUT -j nova-api-OUTPUT -A POSTROUTING -j nova-compute-POSTROUTING -A POSTROUTING -j nova-network-POSTROUTING -A POSTROUTING -j nova-manage-POSTROUTING -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE -A POSTROUTING -j nova-api-POSTROUTING -A POSTROUTING -j nova-postrouting-bottom -A nova-api-snat -j nova-api-float-snat -A nova-compute-snat -j nova-compute-float-snat -A nova-manage-snat -j nova-manage-float-snat -A nova-manage-snat -s 192.168.207.0/24 -j SNAT --to-source 10.131.0.244 -A nova-network-POSTROUTING -s 192.168.200.0/24 -d 10.131.0.244/32 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.200.0/24 -d 10.128.0.0/24 -j ACCEPT -A nova-network-POSTROUTING -s 192.168.200.0/24 -d 192.168.200.0/24 -m conntrack ! --ctstate DNAT -j ACCEPT -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.131.0.244:8775 -A nova-network-snat -j nova-network-float-snat -A nova-network-snat -s 192.168.200.0/24 -j SNAT --to-source 10.131.0.244 -A nova-network-snat -s 8.8.8.0 ... (more)

edit flag offensive delete link more
0

answered 2012-05-17 00:55:17 -0500

danwent gravatar image

Sorry for the delay, I'm really swamped with dev work and questions.

So I was able to reproduce something that seems similar to what you've reported.

I did seem to confirm that the iptables rules are the culprit though, since running "sudo iptables -F" allows the VMs that couldn't ping start to ping.

After some sleuthing I believe it is firewall rules intended to implement a portion of security group filtering that are resulting in the behavior. Disabling the security groups using the following rule in nova.conf worked for me:

firewall_driver=nova.virt.firewall.NoopFirewallDriver

edit flag offensive delete link more
0

answered 2012-05-16 02:21:31 -0500

guestly gravatar image

Do you have any further resolution about this problem?

edit flag offensive delete link more
0

answered 2012-05-15 08:47:05 -0500

mb-s gravatar image

Looking back at this problem... I don't know if this is the reason but here is what I have noticed from looking at the entries in the nat table. Let's say I have a 10.0.0.0/24 public network. After I create a private network, say 10.6.0.0/24 Here are a couple of entries from the nat table:

SNAT all -- 10.6.0.0/24 anywhere to:9.2.156.126 SNAT all -- 10.0.0.0/24 anywhere to:9.2.156.126

This is after creating a new 10.6.0.0 network. Then if I add yet another network (10.8.0.0/24). Here are the table entries:

SNAT all -- 10.8.0.0/24 anywhere to:9.2.156.126 SNAT all -- 10.0.0.0/24 anywhere to:9.2.156.126

As you can see the entry for public network 10.0.0.0 remains but the entry for 10.6 network is replaced by the entry for the newer network. Is this how it should be?

edit flag offensive delete link more
0

answered 2012-04-11 09:08:07 -0500

This question was expired because it remained in the 'Needs information' state without activity for the last 15 days.

edit flag offensive delete link more
0

answered 2012-03-26 15:49:09 -0500

danwent gravatar image

Have you tried running this setup without Quantum and seeing if it works? For Essex, Quantum basically just uses existing Nova code for L3 functionality. If its unique to Quantum, bhall is the best person to look at it, but he has been OOO for the past few days.

What you're doing SHOULD work, but it is a somewhat non-standard config and may not have been tested well. Usually if you were doing a multi-node setup with L3 functionality, nova-network would either be running only on a dedicated controller node, or on each node (with the multi_host flag set to True, which works only for non-Quantum deployments).

edit flag offensive delete link more
0

answered 2012-03-23 19:20:22 -0500

mb-s gravatar image

Just wondering if you found any clues as to what may be going on here? Thanks.

edit flag offensive delete link more
0

answered 2012-03-20 22:17:53 -0500

mb-s gravatar image

Sure.

On Server A:

mb@sysnet45:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=411937.037s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=3 actions=drop cookie=0x0, duration=411934.967s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=2 actions=drop cookie=0x0, duration=411334.842s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=7 actions=drop cookie=0x0, duration=412155.689s, table=0, n_packets=882468, n_bytes=111928515, priority=1 actions=NORMAL

============================================================================

mb@sysnet43:~$ sudo ovs-ofctl dump-flows br-int NXST_FLOW reply (xid=0x4): cookie=0x0, duration=411739.341s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=1 actions=drop cookie=0x0, duration=370942.723s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=5 actions=drop cookie=0x0, duration=412054.101s, table=0, n_packets=582126, n_bytes=71555265,priority=1 actions=NORMAL mb@sysnet43:~$

edit flag offensive delete link more
0

answered 2012-03-20 20:31:54 -0500

bgh gravatar image

one more thing: on both nodes, can you paste the output of: "ovs-ofctl dump-flows br-int" please

edit flag offensive delete link more
0

answered 2012-03-20 19:29:05 -0500

mb-s gravatar image

Sure.

Here is the new data from Server A (9.2.156.126 running all of nova services):

script started on Tue 20 Mar 2012 03:20:17 PM EDT mb@sysnet45:~$ sudo iptables -L -n -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination
nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-network-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-manage-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination
nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
nova-network-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
nova-manage-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain POSTROUTING (policy ACCEPT) target prot opt source destination
nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-network-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-manage-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24

Chain nova-api-OUTPUT (1 references) target prot opt source destination

Chain nova-api-POSTROUTING (1 references) target prot opt source destination

Chain nova-api-PREROUTING (1 references) target prot opt source destination

Chain nova-api-float-snat (1 references) target prot opt source destination

Chain nova-api-snat (1 references) target prot opt source destination
nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-OUTPUT (1 references) target prot opt source destination

Chain nova-compute-POSTROUTING (1 references) target prot opt source destination

Chain nova-compute-PREROUTING (1 references) target prot opt source destination

Chain nova-compute-float-snat (1 references) target prot opt source destination

Chain nova-compute-snat (1 references) target prot opt source destination
nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-manage-OUTPUT (1 references) target prot opt source destination

Chain nova-manage-POSTROUTING (1 references) target prot opt source destination

Chain nova-manage-PREROUTING (1 references) target prot opt source destination

Chain nova-manage-float-snat (1 references) target prot opt source destination

Chain nova-manage-snat (1 references) target prot opt source destination
nova-manage-float-snat all -- 0.0.0.0/0 0.0.0.0/0
SNAT all -- 9.9.9.0/24 0.0.0.0/0 to:9.2.156.126

Chain nova-network-OUTPUT (1 references) target prot opt source destination

Chain nova-network-POSTROUTING (1 references) target prot opt source destination
ACCEPT all -- 10.0.0.0/24 10.128.0.0/24
ACCEPT all -- 10.0.0.0/24 10.0.0 ...

(more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2012-03-16 15:43:03 -0500

Seen: 79 times

Last updated: May 17 '12