Ask Your Question
1

Instances don't get an IP from DHCP (Quantum, OVS, multi-node computes)

asked 2012-05-18 14:45:39 -0500

emilienm gravatar image

Hi,

Since 2 weeks, I've been looking for a solution with a Quantum + OVS issue.

The situation :

2 servers :

Essex-1 - Eth0 : 10.68.1.40 - ETH1 : connected to br-int OVS bridge -> Glance, Nova-*, Keystone, Horizon, Quantum-Server, KVM, OVS, Quantum-Agent -> nova.conf : https://github.com/EmilienM/doc-openstack/blob/master/Configuration%20Files/Essex-1/nova.conf (https://github.com/EmilienM/doc-opens...)

Essex-2 - Eth0 : 10.68.1.45 - ETH1 : connected to br-int OVS bridge -> nova-compute, KVM, Quantum-Agent -> nova.conf : https://github.com/EmilienM/doc-openstack/blob/master/Configuration%20Files/Essex-1/nova.conf (https://github.com/EmilienM/doc-opens...)

I've followed http://openvswitch.org/openstack/documentation/ (http://openvswitch.org/openstack/docu...) and http://docs.openstack.org/trunk/openstack-network/admin/content/ (http://docs.openstack.org/trunk/opens...)

I've created th network with : nova-manage network create --label=mysql --fixed_range_v4=192.168.113.0/24 --project_id=d2f0dc48a8944c6e96cb88c772376f06 --bridge=br-int --bridge_interface=eth1

What's not working : -> When I create an instance from dashboard, the VM does not get an IP from DHCP server (hosted on ESSEX-1). You can see the logs here : http://paste.openstack.org/show/17997/

What I did to investigate : -> dhcpdump -i br-int : I can see DHCPDISCOVER on both servers (without answers) -> ps -ef | grep dnsmasq : nobody 6564 1 0 14:12 ? 00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-gw-0f427a46-3f.pid --listen-address=192.168.113.1 --except-interface=lo --dhcp-range=192.168.113.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-gw-0f427a46-3f.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro root 6565 6564 0 14:12 ? 00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-gw-0f427a46-3f.pid --listen-address=192.168.113.1 --except-interface=lo --dhcp-range=192.168.113.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-gw-0f427a46-3f.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro root 16536 6192 0 14:40 pts/14 00:00:00 grep --color=auto dnsm

Is my nova.conf correct ? What's wrong with my configuration ? Is there a problem with DNSMASQ ?

I would apreciate any idea !

Regards

edit retag flag offensive close merge delete

26 answers

Sort by ยป oldest newest most voted
1

answered 2012-05-28 16:45:44 -0500

emilienm gravatar image

I update with my day work :

Now, my VMs hosted by ESSEX-1 have network, but VMs hosted by ESSEX-2 not.

root@essex-1:/var/lib/nova/networks# more nova-gw-fbbd1249-c5.conf fa:16:3e:5e:e2:f3,host-192.168.113.2.novalocal,192.168.113.2 fa:16:3e:34:bf:36,host-192.168.113.3.novalocal,192.168.113.3

root@essex-2:~# tcpdump -i eth1 | grep DHCP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 16:29:50.653177 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:34:bf:36 (oui Unknown), length 300 16:29:53.315672 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:34:bf:36 (oui Unknown), length 300

For information : fa:16:3e:34:bf:36 is the MAC Adresse of the VM Nic hosted by ESSEX-2.

So, If I resume :

  • The OVS bridge is working
  • DNSMASQ is working (in local that sure, but not with br-int)
  • VMs hosted by ESSEX-1 have network
  • VMs hosted by ESSEX-2 don't have network

Any idea guys ?

edit flag offensive delete link more
0

answered 2012-05-24 17:00:23 -0500

askstack gravatar image

Emilien I have a second node working now. I don't have nova-compute.conf. Both hosts are using nova.conf. http://paste.openstack.org/show/18157/

edit flag offensive delete link more
0

answered 2012-05-18 19:50:56 -0500

askstack gravatar image

I have the same problem as Emilien. tcpdump showes client requesting IP but not getting an answer. dnsmasq is running on the controller. I did not do "ovs-vsctl add-port br-int eth1"

/var/log/message

May 18 15:49:25 core01 ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- --may-exist add-port br-int tap926d429e-53 -- set Interface tap926d429e-53 external-ids:iface-id=926d429e-5313-4d3b-b93a-afe0d99a98b1 -- set Interface tap926d429e-53 external-ids:iface-status=active -- set Interface tap926d429e-53 external-ids:attached-mac=fa:16:3e:2d:bb:a8 -- set Interface tap926d429e-53 external-ids:vm-uuid=b4f94e0f-77bb-4aff-8131-e1f97caf595f May 18 15:49:25 core01 kernel: [92915.553134] device tap926d429e-53 entered promiscuous mode May 18 15:49:25 core01 nova-compute[27734]: 2012-05-18 15:49:25 INFO nova.virt.libvirt.connection [req-167a5e21-2e24-4cb4-8395-13deb6f66155 aa46ceb969f9411494aa2fba527c19e7 76c41e7de2d0408489e94f8adb5b28ee] [instance: b4f94e0f-77bb-4aff-8131-e1f97caf595f] Creating image May 18 15:49:26 core01 kernel: [92916.276732] ADDRCONF(NETDEV_CHANGE): tap926d429e-53: link becomes ready May 18 15:49:27 core01 nova-compute[27734]: 2012-05-18 15:49:27 INFO nova.virt.libvirt.connection [-] [instance: b4f94e0f-77bb-4aff-8131-e1f97caf595f] Instance spawned successfully. May 18 15:49:27 core01 ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl --timeout=2 set Port tap926d429e-53 tag=4 May 18 15:49:30 core01 ntpd[1054]: Listen normally on 35 tap926d429e-53 fe80::e092:c5ff:fe99:20f UDP 123 May 18 15:49:30 core01 ntpd[1054]: peers refreshed May 18 15:49:44 core01 dnsmasq-dhcp[32577]: DHCPDISCOVER(gw-4b4f2f7e-7c) fa:16:3e:2d:bb:a8 no address available May 18 15:49:49 core01 dnsmasq-dhcp[32577]: DHCPDISCOVER(gw-4b4f2f7e-7c) fa:16:3e:2d:bb:a8 no address available May 18 15:49:53 core01 dnsmasq-dhcp[32577]: DHCPDISCOVER(gw-4b4f2f7e-7c) fa:16:3e:2d:bb:a8 no address available

edit flag offensive delete link more
0

answered 2012-05-18 20:38:33 -0500

emilienm gravatar image

Dan,

That's strange :

/var/lib/nova/networks/nova-gw-0f427a46-3f.conf is empty.

And MySQL result :

http://pastebin.com/QF7cgCcg

edit flag offensive delete link more
0

answered 2012-05-21 15:40:00 -0500

danwent gravatar image

there's no need to set --flat_network_bridge to "br-int", but it shouldn't hurt anything.

Based on the database tables you posted, it seems like there is a single running VM with the following IP: 192.168.113.5 . Is that correct, or are there other VMs running?

edit flag offensive delete link more
0

answered 2012-05-21 16:45:20 -0500

emilienm gravatar image

I have one VM in my infrastructure. ( It's not a big cloud yet :-) )

edit flag offensive delete link more
0

answered 2012-05-18 16:24:57 -0500

emilienm gravatar image

Hi Dan,

Thank's for quick answer.

Here my nova-compute.conf for both servers :

--libvirt_type=kvm --libvirt_use_virtio_for_bridges=true --libvirt_ovs_bridge=br-int --libvirt_vif_type=ethernet --libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver

And of course, I did "ovs-vsctl add-port br-int eth1" on both servers.

I use Essex with Ubuntu 12.04 up to date.

If I understand, I have to investigate with tcpdump on all interfaces.

So, what you mean, is it's not possible yet to use multi node compute architecture with Quantum - OVS with VLANS ?

Thank's for your help :-)

Regards

edit flag offensive delete link more
0

answered 2012-05-18 20:40:26 -0500

askstack gravatar image

My /var/lib/nova/networks/nova-gw-4b4f2f7e-7c.conf is empty.

http://paste.openstack.org/show/18026/

edit flag offensive delete link more
0

answered 2012-05-26 18:37:03 -0500

emilienm gravatar image

Thank's askstack, I'll have a look next week, and report here if my issue will be solved.

edit flag offensive delete link more
0

answered 2012-05-18 15:58:31 -0500

danwent gravatar image

Hi Emilian,

Thanks for the detailed report. Two possible causes of the issue jump out to me:

First, for your nova-compute, I don't see you setting the correct vif-driver flags:

libvirt_ovs_bridge=br-int libvirt_vif_type=ethernet libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver

(see "Nova Compute Node Configuration: http://openvswitch.org/openstack/documentation/ (http://openvswitch.org/openstack/docu...) )

Second, Quantum does not pay attention to the --bridge or --bridge_interface options to "nova-manage network create". In theory, this is harmless, but perhaps you're expecting that adding this parameter is linking br-int to the physical network (this is not the case). You will have to add the NIC to br-int as described in the same section of the OVS documentation. For example:

ovs-vsctl add-port br-int eth1

Other than that, I good trick to debug connectivity issues is tcpdump in different locations along the path. This includes tap devices for VMs, physical NICs carrying the traffic to the physical network, and the gw-* devices that dnsmasq binds to.

Btw, it looks like you're using Essex, so VLANS is the your best option. Using OVS in tunnel mode would actually be much easier as you don't need to deal with VLANs on the physical network, but there are a couple bugs in the Essex release for this feature, so until we back-port them, we have not been publicly documenting tunnel mode.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

Stats

Asked: 2012-05-18 14:45:39 -0500

Seen: 485 times

Last updated: Jun 08 '12