VMs not getting IP through DHCP private 10.x network unreachable

asked 2014-11-19 05:54:46 -0500

neel-basu-z
33 ●3 ●3 ●8

updated 2014-11-20 02:22:20 -0500

I am logging in to dashboard through demo user that is having a private network of 10.0.0.0/24. I can see the router and the dhcp namespace in ip netns But I can not ping 10.0.0.1 or 10.0.0.2. I can launch VM but But VM's are not getting IP from DHCP.

The question is long with too many outputs attached please click (more) at the bottom for full question

What I see in the console/log is

Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed
cirros-ds 'net' up at 180.96

VMs are having an eth0 iface that doesn't have an IPv4.

systemctl status neutron-dhcp-agent.service says active(running) but bridge interfaces are DOWN

$ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:22:4d:b1:3b:4f brd ff:ff:ff:ff:ff:ff
    inet 172.16.21.11/23 brd 172.16.21.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::222:4dff:feb1:3b4f/64 scope link 
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether ea:fc:1c:48:e2:57 brd ff:ff:ff:ff:ff:ff
4: br-ex: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 02:09:3f:7c:5b:49 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether 9e:7c:0d:20:16:46 brd ff:ff:ff:ff:ff:ff
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether b6:da:2f:93:dc:85 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
8: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default 
    link/ether aa:94:26:b3:f0:46 brd ff:ff:ff:ff:ff:ff
9: qbr7c65b9a6-a7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 06:73:bb:99:aa:a7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f0c7:e3ff:fea4:17bb/64 scope link 
       valid_lft forever preferred_lft forever
10: qvo7c65b9a6-a7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether 12:a2:36:0e:9d:5c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::10a2:36ff:fe0e:9d5c/64 scope link 
       valid_lft forever preferred_lft forever
11: qvb7c65b9a6-a7: <BROADCAST ...

(more)

edit retag flag offensive close merge delete

Comments

May I ask you if you are running on CentOS7 ?

GLaupre ( 2014-11-19 13:38:31 -0500 )edit

What ps -ef | grep dnsmasq return?

GLaupre ( 2014-11-19 14:00:01 -0500 )edit

View https://ask.openstack.org/en/question...

dbaxps ( 2014-11-20 00:00:16 -0500 )edit

I am running Fedora 20

neel-basu-z ( 2014-11-20 00:04:33 -0500 )edit

ps -ef | grep dnsmasw output https://gist.github.com/anonymous/f3f7338a71d0c82be61a (https://gist.github.com/anonymous/f3f...)

neel-basu-z ( 2014-11-20 00:13:00 -0500 )edit

see more comments

4 answers

Sort by » oldest newest most voted

answered 2014-11-19 09:52:49 -0500

dbaxps

10788 ●57 ●112 ●224 http://bderzhavets.blo...

updated 2014-11-19 12:35:36 -0500

Forced to use answer field , because need formatting.

Run : [root@juno1 ~(keystone_admin)]# neutron net-list
Select yours private-net-id
[root@juno1 ~(keystone_admin)]# ip netns | grep  private-net-id
Say it is qdhcp-45577666-657d-4f75-a3ab-9bc232f15203
Usually tap interface is given by command 
[root@juno1 ~(keystone_admin)]# ip netns exec qdhcp-45577666-657d-4f75-a3ab-9bc232f15203 ip addr show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    13: tap7a12f9b0-a4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
        link/ether fa:16:3e:29:fe:f1 brd ff:ff:ff:ff:ff:ff
        inet 50.0.0.11/24 brd 50.0.0.255 scope global tap7a12f9b0-a4
           valid_lft forever preferred_lft forever
        inet6 fe80::f816:3eff:fe29:fef1/64 scope link 
           valid_lft forever preferred_lft forever

Correspondent tcpdump capturing should look like:-

 [root@juno1 ~(keystone_admin)]# ip netns exec qdhcp-45577666-657d-4f75-a3ab-9bc232f15203 tcpdump -ln -i tap7a12f9b0-a4
........
18:43:31.127181 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:bb:67:1a, length 300
18:43:31.127751 IP 50.0.0.11.bootps > 50.0.0.32.bootpc: BOOTP/DHCP, Reply, length 324
18:43:31.151226 IP 50.0.0.32 > 224.0.0.22: igmp v3 report, 1 group record(s)
18:43:31.399526 IP 50.0.0.32.mdns > 224.0.0.251.mdns: 0 [4q] [7n] ANY (QM)? a.1.7.6.b.b.e.f.f.f.e.3.6.1.8.f.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. ANY (QM)? vf20rsx1511.local. ANY (QM)? 32.0.0.50.in-addr.arpa. ANY (QM)? vf20rsx1511 [fa:16:3e:bb:67:1a]._workstation._tcp.local. (323)
18:43:31.449740 IP 50.0.0.32.mdns > 224.0.0.251.mdns: 0*- [0q] 2/0/0 PTR _workstation._tcp.local., PTR vf20rsx1511 [fa:16:3e:bb:67:1a]._workstation._tcp.local. (118)

Enable also dnsmasq logging and upload somewhere corresponding dnsmasq.log .

Add line
dnsmasq_config_file = /etc/neutron/dnsmasq.conf
to dhcp_agent.ini  && create file /etc/neutron/dnsmasq.conf
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp

dnsmasq.log

Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 DHCPREQUEST(tap7a12f9b0-a4) 50.0.0.32 fa:16:3e:bb:67:1a
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 tags: tag0, known, tap7a12f9b0-a4
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 DHCPACK(tap7a12f9b0-a4) 50.0.0.32 fa:16:3e:bb:67:1a host-50-0-0-32
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 requested options: 1:netmask, 28:broadcast, 2:time-offset, 121:classless-static-route,
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 requested options: 15:domain-name, 6:dns-server, 12:hostname,
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 requested options: 40:nis-domain, 41:nis-server, 42:ntp-server,
Nov 19 18:43:31 dnsmasq-dhcp[5772]: 3802579461 requested options: 26:mtu ...

(more)

edit flag offensive delete link

Comments

Please check edit. I don't see any traffic in tcpdump output. I don't know where to put the lines you mentioned for dnsmasq.log. Please check the other outputs in my question edit. (I think you need to click more. it is truncating the question)

neel-basu-z ( 2014-11-20 00:41:40 -0500 )edit

Add line
dnsmasq_config_file = /etc/neutron/dnsmasq.conf

to /etc/neutron/dhcp_agent.ini

dbaxps ( 2014-11-20 00:44:12 -0500 )edit

Create file /etc/neutron/dnsmasq.conf
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp

Restart neutron services

dbaxps ( 2014-11-20 00:46:09 -0500 )edit

It might be nothing captured by tcpdump on qdhcp-namespace tap-interface, if dnsmasq doesn't work
. That's why I want to get it's log.

dbaxps ( 2014-11-20 00:51:24 -0500 )edit

ps -ef | grep dnsmasq output pasted on gist https://gist.github.com/anonymous/c3fd1d45d88720b19da0 (https://gist.github.com/anonymous/c3f...) after adding those lines and rebooting the machine

neel-basu-z ( 2014-11-20 00:56:03 -0500 )edit

see more comments

answered 2014-11-20 07:13:05 -0500

pixelbeat
543 ●2 ●4 ●16 http://www.pixelbeat.org/

I can't see the version you're using. It's always good to state that.

Please note there was an update to icehouse and juno neutron packages 5 days ago to fix a permissions issue impacting dnsmasq. You should be OK in that regard if you're using >= openstack-neutron-2014.2-9 for Juno, or >= openstack-neutron-2014.1.3-4 for Icehouse

edit flag offensive delete link

Comments

It's OK now. I got feedback on RDO mailing list 11/15/2014 . View https://ask.openstack.org/en/question...

dbaxps ( 2014-11-20 07:23:47 -0500 )edit

add a comment

answered 2014-11-20 21:00:44 -0500

GLaupre

391 ●16 ●22 ●26 http://chelsio.com/

updated 2014-11-20 21:06:40 -0500

Hello,

I had the same issue. My instances received their IPs after I did this: I completely deactivated the iptables and NetowrkingManagement service on all nodes:

[...]    
Starting network...
 udhcpc (v1.20.1) started
 Sending discover...
 Sending select for 10.0.0.20...
 Lease of 10.0.0.20 obtained, lease time 86400
 deleting routers
 adding dns 10.0.0.5
 check-version already run per instance
 Starting dropbear sshd: OK
 userdata already run per instance
[...]

Please do:

1) correct the dnsmasq (running not as nobody) follow the explanations from dbaxps here.
2) stop the iptables and NetworkingManager service
3) Start a new instance and tcpdump the interface on your qdhcp(steps from uckey here)

If your new instance gets its IP, that's great, but in this case, maybe there is a bug to be reported about the default rules in the iptables, no? The instances should be able to talk with the qdhcp without any rules added in the iptables by hand, I guess..

I am waiting for your answer.

edit flag offensive delete link

Comments

Please, view https://bugzilla.redhat.com/show_bug....
If 1) correct the dnsmasq (running not as nobody) helps, then official fix

 # chmod o+rx /var/lib/neutron
 # chmod o+rx /var/log/neutron

dbaxps ( 2014-11-21 00:21:31 -0500 )edit

This didn't work for me, dbaxps. I had to try something else (I am using devstack)

GLaupre ( 2014-11-21 11:20:01 -0500 )edit

add a comment

answered 2014-11-19 08:32:18 -0500

uckey
51 ●1 ●3

updated 2014-11-19 08:39:29 -0500

You should confirm whether dhcp request is sent from vm to Network Node on which dhcp is running or not.

By the way , you can see dhcp packet which is sent from vm with following command .

(1) Confirm dhcp network namespace

$sudo ip netns

output

dhcp*****

(2)Confirm nic which dhcp namespace have

$sudo ip netns exec <dhcp*****> ip addr show

output

eth0:*****

eth1:*****

(3) Cofirm dhcp packet using tcpdump

$sudo ip netns exec <dhcp****> tcpdump -i <eth0 or eth1>

If you don't mind upload the result you did above step , Plead upload one.

edit flag offensive delete link

Comments

Please check the edit in my question. I don't see any traffic in tcpdump output. I've added other outputs in edit too.

neel-basu-z ( 2014-11-20 00:42:47 -0500 )edit

also I cannot ping 10.0.0.1 from 10.0.0.2. Please check the edit in question

neel-basu-z ( 2014-11-20 03:07:31 -0500 )edit

add a comment

VMs not getting IP through DHCP private 10.x network unreachable

Comments

4 answers

Comments

Comments

Comments

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

VMs not getting IP through DHCP private 10.x network unreachable edit

Comments

4 answers

Comments

Comments

Comments

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

VMs not getting IP through DHCP private 10.x network unreachable