Ask Your Question

How does Neutron form the full mesh of GRE tunnels?

asked 2013-08-31 13:45:15 -0600

nko321 gravatar image

updated 2013-09-05 13:09:33 -0600

darragh-oreilly gravatar image


I'm having some trouble understanding how to use Neutron to connect my VMs to the outside network within my OpenStack test environment. I’m configuring this environment using these instructions:

They are scant on details regarding how to configure my physical switches to support the interface configurations listed for each node and outlined in Section 1: Requirements. It is my understanding that VMs on the Compute Node have all of their networking behind br-int, which then talks over a GRE tunnel to the Network Node. The GRE tunnel appears to use the VM Conf Network as described in the document.

However, I don’t see anywhere in the document where the Network Node is declared to be an endpoint in any GRE tunnels. I’m a bit under-experienced with networking, but I would assume that the Compute Node would need to be told specifically where to connect the GRE tunnel to. I see that both the Compute Node and Network Node have a Local IP specified in their Neutron configurations, but I don’t see anywhere that would tell them to take traffic from the VMs and shove it into the Network Node.

This brings us to the problem I’m having: after following all of the instructions in the above-linked document, I am able to:

  • Log into Horizon.

  • Create tenants (projects), users, flavors, images, instances, networks, routers, ports, floating IPs, security groups, keypairs, see my appropriate-looking network topology… every command executes successfully and behaves as expected.

  • See and use the Console for each VM.

However, I can’t get my VMs to connect to anything. They won’t connect to my (physical) router’s IP, they won’t connect beyond their own subnet… they can ping and SSH and everything to each other so long as they’re on the same subnet, but nothing else.

This leads me to suspect that my host environment’s network configuration is wrong and I specifically suspect my Neutron configuration and its GRE tunnel.

I have tons of questions I’d like to ask but really, the problem is that I don’t understand how this connectivity is supposed to flow, where exactly it’s all configured (I actually think I DO understand WHERE to configure it, but I must be missing a specific parameter or something), etc. etc., and I’ve yet to come across any documentation that imparted an understanding of these things upon me.

Could you take a couple minutes to dumb it down for me? The traffic flow I’m aiming for, the GRE config, how to check for the pertinent parts, etc.? I would SUPER appreciate any bit of wisdom you could spare.

Thanks for reading!!!!

edit retag flag offensive close merge delete

2 answers

Sort by » oldest newest most voted

answered 2013-09-02 08:43:59 -0600

darragh-oreilly gravatar image

You can check your tunnels by running sudo ovs-vsctl show on the nodes. There should be a port on br-tun for every other node running the ovs-agent.

The full mesh of tunnels is kept in sync using rpc here and here.

edit flag offensive delete link more


Thanks!! I see that my compute node does have a GRE interface with my network node and vice versa. That's an excellent thing to be able to know for sure and it gives me something to go on!

nko321 gravatar imagenko321 ( 2013-09-02 12:06:43 -0600 )edit

ok, that figures, as your instance did get an ip from the dhcp server which runs on the network node. I can't guess what's wrong with your quantum router. Maybe you could follow my blog post about how it works

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-09-02 12:53:10 -0600 )edit

Thanks- I'm reading now. By the way, somehow, an extra "< / p >" got added to the end of that link.

nko321 gravatar imagenko321 ( 2013-09-02 13:05:33 -0600 )edit

I believe I've got a much, much better understanding of what's going on now. My OVS network is performing perfectly except my external network's router can't see the gateway on the external network. My bridge config appears healthy, but I'm about to run out of characters how do I talk more on forum?

nko321 gravatar imagenko321 ( 2013-09-04 17:30:33 -0600 )edit

That's good. You can append to your question: 'quantum net-show {ext-net-id}' and 'quantum subnet-show {ext-net-subid}' and 'ip -4 address' and 'ip netns exec qrouter-{uuid} ip -4 a' and 'ip netns exec qrouter-{uuid} route -n' and 'ovs-vsctl show'. Or better, open a new question for this.

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-09-05 04:25:52 -0600 )edit

answered 2013-09-12 11:04:46 -0600

slogan621 gravatar image

Short answer is the GRE mesh is created because at start up, the OVS agent on a compute node sends RPCs to the controller to solicit tunnel notifications. As a result, the controller will create a tunnel to the compute node (using the compute node's IP address as the end point), and tell all other compute nodes about the compute node, causing them to create tunnels too. The compute node at this time will also be sent a list of all of the tunnels it needs to create. The agent then forever sits in a loop, listing for notifications about other compute nodes leaving and joining the cluster via tunnel notifications. For each notification, it will create (and destroy) tunnels to these nodes as a result. Simple algorithm, actually.

Here's my basic flow on the command line for starting and pinging nodes in a GRE-meshed network. I'm not a horizon user (ironic, since I do a lot of Django development, but oh well :-) but perhaps there is an equivalent in horizon. I don't think there is anything keeping you from using both horizon and command line to experiment with things before you go looking for a horizon-only solution The key takeaway is the need to configure security groups to allow pings and to issue the pings from within a namespace.

On the controller:

$ nova secgroup-add-rule default tcp 22 22
| IP Protocol | From Port | To Port | IP Range  | Source Group |
| tcp         | 22        | 22      | |              |
$ nova secgroup-add-rule default icmp -1 -1
| IP Protocol | From Port | To Port | IP Range  | Source Group |
| icmp        | -1        | -1      | |              |

The above enables both ping and ssh. I do both because ssh'ing into VMs is something I do a lot of, but you probably only have to add the icmp rule for ping.

The rest of the flow:

$ nova image-list
| ID                                   | Name                            | Status | Server |
| 651d944f-a532-43ee-b938-2682a6c66159 | cirros-0.3.1-x86_64-uec         | ACTIVE |        |
| 29290504-b981-49c1-8641-0ba14643c73a | cirros-0.3.1-x86_64-uec-kernel  | ACTIVE |        |
| b972d37e-e09d-42e6-8967-511f62ce064b | cirros-0.3.1-x86_64-uec-ramdisk | ACTIVE |        |

$  neutron net-list
| id                                   | name    | subnets                                          |
| ee03906b-e70e-45de-9a7b-6dbbc5b38916 | private | 4e5838f4-9c51-4301-9d55-896fcdb5ad47 |

$ nova boot --image 651d944f-a532-43ee-b938-2682a6c66159 --flavor 1 --nic net-id=ee03906b-e70e-45de-9a7b-6dbbc5b38916 test6

<wait 10="" addr="" an="" and="" for="" get="" here="" ip="" let="" seconds="" spin="" the="" to="" up="" vm="">

$ nova list
| ID                                   | Name  | Status | Task State | Power State | Networks         |
| e87f3686-82ed-4048-9c4f-ff3bcfc578f6 | test1 | ACTIVE | None       | Running     | private= |
| faed2ef9-c6a4-4b4b-87cd-e027e1d81f57 | test2 | ACTIVE | None       | Running     | private= |
| 2642d183-d687-42f4-918e-c42953a0723c | test3 | ACTIVE | None       | Running     | private= |
| f9763c59-fca7-4e79-962b-6d8b592e22c5 | test4 | ACTIVE | None       | Running     | private= |
| 41ae9116-47cd-4a93-a1de-0164e386bad3 | test5 | ACTIVE | None       | Running     | private= |
| 3d86d3ed-cfd5-4fb5-8629-4f4504516d1b | test6 | ACTIVE | None       | Running     | private= |

$ ip netns list

$ sudo ip netns exec qdhcp-ee03906b-e70e-45de-9a7b-6dbbc5b38916 bash
# ping
64 bytes from icmp_req=1 ttl=64 time=1.77 ms
64 bytes from icmp_req=2 ttl=64 time=0.527 ms
64 ...
edit flag offensive delete link more


By the way, you can see these tunnel notifications by filtering AMQP traffic in wireshark.

slogan621 gravatar imageslogan621 ( 2013-09-12 11:06:12 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2013-08-31 13:45:15 -0600

Seen: 2,508 times

Last updated: Sep 12 '13