Here is iptables from my compute-node. But I don't understand why the compute-node should be an suspect for my problem? If the network-node would be working correctly I should see the dhcp reply packet on br-int on my network-node, shouldn't I?

I can see it on device tap66178edc-18 which is member of br-int. So why can I see the packet on that tap device but not on the bridge it is connected to? It does work the other way round. Packets I can see on br-int, I can also see on the tap-device.

That fact makes me believe that the root of that problem lies in OVS. But I'm really new to OpenStack, so please let me know if I misunderstand here anything.

root@compute1:~# iptables-save -c

Generated by iptables-save v1.4.12 on Mon Apr 15 12:02:24 2013

*nat :PREROUTING ACCEPT [52447:9171960] :INPUT ACCEPT [20:1332] :OUTPUT ACCEPT [156238:9580558] :POSTROUTING ACCEPT [208665:18751186] :nova-compute-OUTPUT - [0:0] :nova-compute-POSTROUTING - [0:0] :nova-compute-PREROUTING - [0:0] :nova-compute-float-snat - [0:0] :nova-compute-snat - [0:0] :nova-postrouting-bottom - [0:0] [52106:9123565] -A PREROUTING -j nova-compute-PREROUTING [154492:9463387] -A OUTPUT -j nova-compute-OUTPUT [206590:18586312] -A POSTROUTING -j nova-compute-POSTROUTING [207841:18692599] -A POSTROUTING -j nova-postrouting-bottom [206590:18586312] -A nova-compute-snat -j nova-compute-float-snat [206590:18586312] -A nova-postrouting-bottom -j nova-compute-snat COMMIT

Completed on Mon Apr 15 12:02:24 2013

Generated by iptables-save v1.4.12 on Mon Apr 15 12:02:24 2013

*mangle :PREROUTING ACCEPT [3325771:2192960480] :INPUT ACCEPT [3212526:2171644761] :FORWARD ACCEPT [382617:103977435] :OUTPUT ACCEPT [3700045:1240719998] :POSTROUTING ACCEPT [4082662:1344697433] COMMIT

Completed on Mon Apr 15 12:02:24 2013

Generated by iptables-save v1.4.12 on Mon Apr 15 12:02:24 2013

*filter :INPUT ACCEPT [3035823:2121673103] :FORWARD ACCEPT [357777:95580751] :OUTPUT ACCEPT [3690400:1239848204] :nova-compute-FORWARD - [0:0] :nova-compute-INPUT - [0:0] :nova-compute-OUTPUT - [0:0] :nova-compute-inst-3 - [0:0] :nova-compute-inst-4 - [0:0] :nova-compute-local - [0:0] :nova-compute-provider - [0:0] :nova-compute-sg-fallback - [0:0] :nova-filter-top - [0:0] [3161946:1874674482] -A INPUT -j nova-compute-INPUT [160848:31011194] -A INPUT -p gre -j ACCEPT [382617:103977435] -A FORWARD -j nova-filter-top [356806:95416570] -A FORWARD -j nova-compute-FORWARD [3672308:1233787427] -A OUTPUT -j nova-filter-top [3652067:1227608280] -A OUTPUT -j nova-compute-OUTPUT [0:0] -A nova-compute-inst-3 -m state --state INVALID -j DROP [13905:4635628] -A nova-compute-inst-3 -m state --state RELATED,ESTABLISHED -j ACCEPT [20:2612] -A nova-compute-inst-3 -j nova-compute-provider [4:1364] -A nova-compute-inst-3 -s 10.5.5.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT [10:768] -A nova-compute-inst-3 -s 10.5.5.0/24 -j ACCEPT [1:60] -A nova-compute-inst-3 -p tcp -m tcp --dport 22 -j ACCEPT [5:420] -A nova-compute-inst-3 -p icmp -j ACCEPT [0:0] -A nova-compute-inst-3 -j nova-compute-sg-fallback [0:0] -A nova-compute-inst-4 -m state --state INVALID -j DROP [2:168] -A nova-compute-inst-4 -m state --state RELATED,ESTABLISHED -j ACCEPT [10710:3704874] -A nova-compute-inst-4 -j nova-compute-provider [10707:3704622] -A nova-compute-inst-4 -s 10.5.5.2/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT [3:252] -A nova-compute-inst-4 -s 10 ... (more)

Hi Darragh,

No, it's not my intention to mix it. What can I do to fix this. I followed the basic install guide for ubuntu 12.04 http://docs.openstack.org/folsom/basic-install/content/basic-install_intro.html (http://docs.openstack.org/folsom/basi...)

As suggested I used the packages from :

root@network:~# cat /etc/apt/sources.list.d/cloud-archive.list deb http://ubuntu-cloud.archive.canonical.com/ubuntu (http://ubuntu-cloud.archive.canonical...) precise-updates/folsom main

Currently installed on network-node:

root@network:~# dpkg -l|grep -i cloud0 ii python-cliff 1.1.2-0ubuntu2~cloud0 Command Line Interface Formulation Framework ii python-eventlet 0.9.17-0ubuntu1.1~cloud0 concurrent networking library for Python ii python-greenlet 0.3.3-1ubuntu2~cloud0 Lightweight in-process concurrent programming ii python-keystone 2012.2.1-0ubuntu1.3~cloud0 OpenStack identity service - Python library ii python-keystoneclient 1:0.1.3-0ubuntu1.1~cloud0 Client libary for Openstack Keystone API ii python-prettytable 0.6-1ubuntu1~cloud0 library to represent tabular data in visually appealing ASCII tables ii python-quantum 2012.2.1-0ubuntu1~cloud0 Quantum is a virutal network service for Openstack. (python library) ii python-quantumclient 1:2.1-0ubuntu1~cloud0 client - Quantum is a virtual network service for Openstack ii python-sqlalchemy 0.7.8-1ubuntu1~cloud0 SQL toolkit and Object Relational Mapper for Python ii python-sqlalchemy-ext 0.7.8-1ubuntu1~cloud0 SQL toolkit and Object Relational Mapper for Python - C extension ii quantum-common 2012.2.1-0ubuntu1~cloud0 common - Quantum is a virtual network service for Openstack. ii quantum-dhcp-agent 2012.2.1-0ubuntu1~cloud0 Quantum is a virtual network service for Openstack. (dhcp agent) ii quantum-l3-agent 2012.2.1-0ubuntu1~cloud0 Quantum is a virtual network service for Openstack. (l3 agent) ii quantum-plugin-openvswitch 2012.2.1-0ubuntu1~cloud0 Quantum is a virtual network service for Openstack. (openvswitch plugin) ii quantum-plugin-openvswitch-agent 2012.2.1-0ubuntu1~cloud0 Quantum is a virtual network service for Openstack. (openvswitch plugin agent) root@network:~#

And on the compute-node:

root@compute1:~# dpkg -l|grep -i cloud0 ii librados2 0.48.2-0ubuntu2~cloud0 RADOS distributed object store client library ii librbd1 0.48.2-0ubuntu2~cloud0 RADOS block device client library ii libvirt-bin 0.9.13-0ubuntu12.2~cloud0 programs for the libvirt library ii libvirt0 0.9.13-0ubuntu12.2~cloud0 library for interfacing with different virtualization systems ii nova-common 2012.2.1+stable-20121212-a99a802e-0ubuntu1.4~cloud0 OpenStack Compute - common files ii nova-compute 2012.2.1+stable-20121212-a99a802e-0ubuntu1.4~cloud0 OpenStack Compute - compute node ii nova-compute-kvm 2012.2.1+stable-20121212-a99a802e-0ubuntu1.4~cloud0 OpenStack Compute - compute node (KVM) ii python-cinderclient 1:1.0.0-0ubuntu1~cloud0 python bindings to the OpenStack Volume API ii python-cliff 1.1.2-0ubuntu2~cloud0 Command Line Interface Formulation Framework ii python-eventlet 0.9.17-0ubuntu1.1~cloud0 concurrent networking library for Python ii python-glance 2012.2.1-0ubuntu1.2~cloud0 OpenStack Image Registry and Delivery Service - Python library ii python-glanceclient 1:0.5.1-0ubuntu1~cloud0 Client library for Openstack glance server. ii python-greenlet 0.3.3-1ubuntu2~cloud0 Lightweight in-process concurrent programming ii python-jsonschema 0.2-1ubuntu1~cloud0 An(other) implementation of JSON Schema (Draft 3) for Python ii python-keystone 2012.2.1-0ubuntu1.3~cloud0 OpenStack identity service - Python ... (more)

Hi Darragh,

Thanks for that hint. Instance isn't getting an ip yet, but it's getting warmer. I now do the capturing on the physical devices, so that I won't disturb the virtual switches. What I can see so far is, that dhcp offers make it from the compute- to the network-node and also back. I captured on eth1 on both nodes as those interfaces are connected to my data lan:

network-node:

root@network:~# tcpdump -nn -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 16:15:37.546449 IP 10.10.10.233 > 10.10.10.232: GREv0, key=0x1, length 354: IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:ac:91:66, length 300 16:15:37.547182 IP 10.10.10.232 > 10.10.10.233: GREv0, key=0x0, length 362: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 308 16:15:42.559841 IP 10.10.10.232 > 10.10.10.233: GREv0, key=0x0, length 54: ARP, Request who-has 10.5.5.3 tell 10.5.5.2, length 28

So, dhcp request and offer packets come and go through my GRE tunnel. That looks good for me. I can see the corresponding packets on my compute-node as well:

root@compute1:~# tcpdump -nn -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 16:15:09.561187 IP 10.10.10.233 > 10.10.10.232: GREv0, key=0x1, length 354: IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:ac:91:66, length 300 16:15:09.562266 IP 10.10.10.232 > 10.10.10.233: GREv0, key=0x0, length 362: IP 10.5.5.2.67 > 10.5.5.3.68: BOOTP/DHCP, Reply, length 308

So the dhcp-offer reaches the compute-node, but doesn't make it back to the instance's network interface. BTW: Do you know a document where is described why an instance creates 4 network devices and how they work together?

root@compute1:~# tcpdump -nn -i vnet0 tcpdump: WARNING: vnet0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vnet0, link-type EN10MB (Ethernet), capture size 65535 bytes 16:15:37.546014 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:ac:91:66, length 300 16:15:37.546067 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:ac:91:66, length 300 16:15:51.320523 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:ac:91:66, length 300 16:15:51.320580 ... (more)