Ask Your Question

Running a router inside an OpenStack VM

asked 2016-07-17 02:16:07 -0500

shimi gravatar image

I'm trying to make a router VM (actually an IPsec VPN server) inside OpenStack. The OpenStack is an installation of Mirantis 7 (why not newer? 8 from some reason causes kernel OOPS to the same image that works on 7, and 9 that was just released... not tested yet).

We're using a Ceph-based installation and "Neutron with tunnelling segmentation".

Looking inside a VM at the output of 'arp -n' shows that it sees MAC addresses of other VMs in its network, so I have imagined that a simple 'route add gw [other VM IP]' in Linux, then pinging would result in those packets appearing on tcpdump running on [other VM IP] output. That does not seem to be the case.

Googling a bit, I found this document: ( - which sounds relevant (I have a similar behaviour in GCE - which is solvable simply by creating VMs with a "IP Forwarding" flag enabled) - so I tried that (or at least I think I did, the instructions assume I know the precise steps, while I am merely guessing), by editing /etc/neutron/plugins/ml2/ml2_conf.ini on each and every one of the controllers in my Mirantis cluster, and adding the line:

extension_drivers = port_security

and then executing:

service neutron-server restart

which, by the way, did not have any affect at the beginning. I am assuming that maybe just one of the 3 is primary (active and 2 standby's), and as long as one with the old config was running, that kept things alive.

When I did the same on the last controller - all hell broke lose - and API calls to get the network status (openstack ... network show [netID]) returned that the API returned an error.

Then I started examining the logs, I found out that the below exception was thrown every time I tried the API call:

<166>Jul 11 16:49:35 node-25 neutron-metadata-agent 2016-07-11 16:49:35.993 17687 INFO eventlet.wsgi.server [-] (17687) accepted ''
<163>Jul 11 16:49:36 node-25 neutron-metadata-agent 2016-07-11 16:49:36.074 17687 ERROR neutron.agent.metadata.agent [-] Unexpected error.
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/", line 109, in __call__
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/", line 216, in _get_instance_and_tenant_id
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent     ports = self._get_ports(remote_address, network_id, router_id)
2016-07-11 16:49:36.074 17687 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/", line 204, in _get_ports
2016-07-11 16:49:36 ...
edit retag flag offensive close merge delete


1 answer

Sort by ยป oldest newest most voted

answered 2016-07-18 16:27:45 -0500

for running a router VM inside your tenant you should disable anti-spoofing on veth, and you should instruct your neutron router what is the next hop for your "internal" network.

For example if you have your network called net-external where your neutron router has an interface at ip you could put your VM to ip Next you can have an internal network called net-internal where your VMROUTER is the default GW with IP

So this could the steps:

neutron net-create net-external
neutron subnet-create --name subnet_external --allocation-pool start=,end= net-external

Create a router01 and attach a new interface to the subnet_external -> you can do this on horizon

neutron net-create net-internal
neutron subnet-create --name subnet_internal --allocation-pool start=,end= --gateway net-internal

neutron security-group-create --description 'A permissive security group to be applied to the gateway' gateway-security-group
neutron security-group-rule-create --direction ingress --remote_ip_prefix gateway-security-group

create the internal port:

neutron port-create --name internal_gw_port --fixed-ip ip_address= --security-group gateway-security-group net-internal

Now that's the trick! disable anti-spoofing to the internal subnet (you must change mac address):

neutron port-update internal_gw_port --allowed_address_pairs type=dict list=true mac_address=fa:16:3e:8d:69:50,ip_address=

Now we create the port for the external subnet:

neutron port-create --name external_gw_port --fixed-ip ip_address= --security-group gateway-security-group net-external

and also we have to permit packets (no anti-spoofing) for the subnet_internal ( on the external veth -> port external_gw_port (you must change mac address):

neutron port-update external_gw_port --allowed_address_pairs type=dict list=true mac_address=fa:16:3e:25:69:92,ip_address=

Now you have to insert on your neutron router a static route to your subnet_internal -> neutron doesn't have access to this subnet, and it must forward packets to your VM

neutron router-update router01 --routes type=dict list=true nexthop=,destination=

Now you can boot your router VM and pass to it the 2 ports (you must change the port-id):

nova boot --flavor m1.small --key-name "YOUR KEY" --image YOUR_ROUTER_IMAGE --nic port-id=c95b4f6c-2ac5-405a-a532-bd6f7e299a73 --nic port-id=190f4b1b-eecf-483d-b156-3a66f1a4a836 --config-drive=true VMROUTER

Another little trick: if you wanna to use a floating ip on your instances besides your VMROUTER (on subnet_internal) you have to assign a multiple floating IPs to your VMROUTER on the same port, and to do this, you have to assign multiple "private" IPs on external_gw_port; for example to add the IP

neutron port-update external_gw_port --fixed-ip subnet_id=ad19756e-2652-4e8f-a0fd-5dc3b0835070,ip_address= --fixed-ip subnet_id=ad19756e-2652-4e8f-a0fd-5dc3b0835070,ip_address=

on the above example you could use as a primary IP and the IP as a secondary IP ... (more)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2016-07-17 02:16:07 -0500

Seen: 995 times

Last updated: Jul 18 '16