So I've got 3 physical machines setup running Nova + Neutron. Each machine has 3 NICs:

1. Openstack backend NIC - eth0 (Statically ssigned IPs on br-ext, this is attached to our normal local office network on the same subnet)
2. Tenant network NIC - eth1
3. Public/Internet NIC - eth2

I'm using a Cisco 3560G switch. eth0 and eth2 ports are setup like this:

interface GigabitEthernet0/1

switchport access vlan 900 (or 902 for eth2)

switchport mode access

speed 1000

duplex full

storm-control broadcast level 5.00

eth1 ports are setup like this:

interface GigabitEthernet0/29

switchport access vlan 901

switchport trunk encapsulation dot1q

switchport trunk native vlan 901

switchport trunk allowed vlan 1-1000

switchport mode trunk

storm-control broadcast level 5.00

!

I have plugged two laptops into the eth1 ports (tenant networks), setup vlan tagging, and confirmed that they can talk to each other on whatever NIC-configured vlan.

My Neutron.conf: http://pastebin.com/pjf8nTF2

My l3_agent.ini: http://pastebin.com/9zauYyK6

My dhcp_agent.ini: http://pastebin.com/VM9LZF5M

My plugin.ini (ml2_conf.ini): http://pastebin.com/F274ZwHZ

Neutron net-list: http://pastebin.com/XHi0XH5L (10.10.9.0/24 is on VLAN 401)

Neutron subnet-list: http://pastebin.com/03SqcTGk

ovs-vsctl show on controller (hosting compute nodes as well): http://pastebin.com/DuPfYNhz

ovs-vsclt show on compute node #1: http://pastebin.com/sJLeP6nh

ovs-vsctl show on compute node #2: http://pastebin.com/a8AppTAJ

Here's the problem:

1. A instance can only get a DHCP'ed if that instance's host (and only that host) is running neutron-dhcp-agent on the same machine.

2. Only physical machine's instances can ping the gateway and access the internet. All instance's can ping other instances on the same node.

My Questions:

1. Am I approaching this correctly? Does my network setup make sense to the casual observer?

2. Does neutron-server, l3-agent, dhcp-agent, openvswitch-agent need to be running on all compute nodes?

3. Does it matter where you run neutron router-gateway-set? It seems to create the routing bridge on the exact machine you run it on.

Any help is greatly appreciated!