Ask Your Question
0

dhcp issue RDO AIO juno

asked 2014-12-09 04:11:02 -0600

updated 2014-12-09 04:22:26 -0600

Hi, I correctly installed Juno release with RDO on CentOS7 machine. Here some reference on used versions:

[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# rpm -qa | grep neutron
python-neutron-2014.2-11.el7.centos.noarch
openstack-neutron-2014.2-11.el7.centos.noarch
openstack-neutron-openvswitch-2014.2-11.el7.centos.noarch
python-neutronclient-2.3.9-1.el7.centos.noarch
openstack-neutron-metering-agent-2014.2-11.el7.centos.noarch
openstack-neutron-ml2-2014.2-11.el7.centos.noarch

Now I am facing a problem when starting VMs, seems dhcp is not working fine; here some debug commands:

[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# ll
total 20
-rw-r--r--. 1 neutron neutron 156 Dec  9 10:42 addn_hosts
-rw-r--r--. 1 neutron neutron 168 Dec  9 10:42 host
-rw-r--r--. 1 neutron neutron  14 Dec  9 10:42 interface
-rw-r--r--. 1 neutron neutron  67 Dec  9 10:42 opts
-rw-r--r--. 1 root    root      6 Dec  9 10:42 pid
[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# cat addn_hosts
10.0.0.2        host-10-0-0-2.openstacklocal host-10-0-0-2
10.0.0.9        host-10-0-0-9.openstacklocal host-10-0-0-9
10.0.0.1        host-10-0-0-1.openstacklocal host-10-0-0-1
[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# cat host
fa:16:3e:17:69:78,host-10-0-0-2.openstacklocal,10.0.0.2
fa:16:3e:2b:ea:63,host-10-0-0-9.openstacklocal,10.0.0.9
fa:16:3e:8a:3e:fc,host-10-0-0-1.openstacklocal,10.0.0.1
[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# cat interface
tap3c06f7ab-47[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# cat opts
tag:tag0,option:router,10.0.0.1
tag:tag0,option:dns-server,10.0.0.2[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# cat pid
11846
[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# ps -ef | grep dnsmasq
nobody   11846     1  0 10:42 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap3c06f7ab-47 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/6137f6d8-ee0b-451c-ac69-7178b6964f9c/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/6137f6d8-ee0b-451c-ac69-7178b6964f9c/host --addn-hosts=/var/lib/neutron/dhcp/6137f6d8-ee0b-451c-ac69-7178b6964f9c/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/6137f6d8-ee0b-451c-ac69-7178b6964f9c/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal
root     13774  3865  0 10:54 pts/0    00:00:00 grep --color=auto dnsmasq
[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# ip netns exec qdhcp-6137f6d8-ee0b-451c-ac69-7178b6964f9c ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
9: tap3c06f7ab-47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether fa:16:3e:17:69:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global tap3c06f7ab-47
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe17:6978/64 scope link
       valid_lft forever preferred_lft forever

I also followed tips from dbaxps in this answer but it doesn't help.

No tcpdump capturing line when performing:

[root@juno61-60-controller 6137f6d8-ee0b-451c-ac69-7178b6964f9c]# ip netns exec qdhcp-6137f6d8-ee0b-451c-ac69-7178b6964f9c tcpdump -ln -i tap3c06f7ab-47
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap3c06f7ab-47, link-type EN10MB (Ethernet), capture size 65535 bytes

Is it correct that 'dnsmasq' process is owned by 'nobody' user? I also tried running ... (more)

edit retag flag offensive close merge delete

Comments

What is in dnsmasq.log ?

dbaxps gravatar imagedbaxps ( 2014-12-09 04:15:04 -0600 )edit

Do I have to specifically enable it? I tried adding log-facility = /var/log/neutron/dnsmasq.log and log-dhcp in dnsmasq-neutron.conf but no log file is created restarting neutron-dhcp-agent service

Antonio G. gravatar imageAntonio G. ( 2014-12-09 04:33:45 -0600 )edit

PS - dnsmasq_config_file =/etc/neutron/dnsmasq-neutron.conf set in dhcp_agent.ini

Antonio G. gravatar imageAntonio G. ( 2014-12-09 04:36:22 -0600 )edit

You do have enable dnsmasq.log. The easiest way to troubleshoot issue. Add to dhcp_agent.ini :

dnsmasq_config_file = /etc/neutron/dnsmasq.conf
dbaxps gravatar imagedbaxps ( 2014-12-09 04:37:56 -0600 )edit

Under /etc/neutron create dnsmasq.conf :

log-facility = /var/log/neutron/dnsmasq.log
log-dhcp
dbaxps gravatar imagedbaxps ( 2014-12-09 04:39:48 -0600 )edit

3 answers

Sort by » oldest newest most voted
0

answered 2014-12-09 04:56:58 -0600

updated 2014-12-09 09:33:49 -0600

Writing here for space and formatting issues... it is a comment!

Ok, now I see this error:

[root@juno61-60-controller log]# service neutron-dhcp-agent status
Redirecting to /bin/systemctl status  neutron-dhcp-agent.service
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: active (running) since Tue 2014-12-09 11:41:38 CET; 12min ago
 Main PID: 4390 (neutron-dhcp-ag)
   CGroup: /system.slice/neutron-dhcp-agent.service
           └─4390 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf -...

Dec 09 11:53:07 juno61-60-controller.cselt.it sudo[7658]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro...terface
Dec 09 11:53:07 juno61-60-controller.cselt.it dnsmasq[7660]: cannot open log /var/log/neutron/dnsmasq.log: Permission denied
Dec 09 11:53:07 juno61-60-controller.cselt.it dnsmasq[7660]: FAILED to start up
Dec 09 11:53:37 juno61-60-controller.cselt.it sudo[7760]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro...b-47 up
Dec 09 11:53:37 juno61-60-controller.cselt.it sudo[7763]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro... global
Dec 09 11:53:37 juno61-60-controller.cselt.it sudo[7766]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro...pe link
Dec 09 11:53:37 juno61-60-controller.cselt.it sudo[7769]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro...f7ab-47
Dec 09 11:53:37 juno61-60-controller.cselt.it sudo[7772]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap /etc/neutron/ro...terface
Dec 09 11:53:38 juno61-60-controller.cselt.it dnsmasq[7774]: cannot open log /var/log/neutron/dnsmasq.log: Permission denied
Dec 09 11:53:38 juno61-60-controller.cselt.it dnsmasq[7774]: FAILED to start up
Hint: Some lines were ellipsized, use -l to show in full.

Also tried setting:

[root@juno61-60-controller log]# chmod 777 /var/log/neutron/

Nothing changes...

FINAL SOLUTION

That's weird... seems to be an issue related to demo subnet created automatically by RDO. If I manually create another private network, connected to public one by a new router, the dhcp on that network correctly works.

I noticed in Horizon that the port associated with the dhcp on the correctly working subnet is in ACTIVE Status while the not working one is in DOWN status. Here CLI reference commands that show this strange behavior.

Does anyone know if it is the expected behavior for RDO install on fresh CentOS7 with default demo tenant activated through answer file?

edit flag offensive delete link more

Comments

So , after

# chmod o+rx /var/lib/neutron
# chmod o+rx /var/log/neutron
dbaxps gravatar imagedbaxps ( 2014-12-09 05:05:49 -0600 )edit

neutron-dhcp-agent fails to start ? Is it correct ?

dbaxps gravatar imagedbaxps ( 2014-12-09 05:07:46 -0600 )edit

yes, it is correct... neutron-dhcp-agent fails to start if I enable logging as you suggested. I could anly see this error cannot open log /var/log/neutron/dnsmasq.log: Permission denied when performing service neutron-dhcp-agent status

Antonio G. gravatar imageAntonio G. ( 2014-12-09 05:19:21 -0600 )edit

Have you disabled SELINUX ?

dbaxps gravatar imagedbaxps ( 2014-12-09 05:19:39 -0600 )edit

you were right, I forgot to set it to permissive through setenforce 0

Antonio G. gravatar imageAntonio G. ( 2014-12-09 07:12:20 -0600 )edit
1

answered 2014-12-10 06:27:24 -0600

DanIzack gravatar image

updated 2014-12-10 07:40:12 -0600

I think it's very hard to predict packstack --allinone or packstack behavior during multinode deployment when SELINUX is enabled. It might be unexpected (weird as wrote Antonio ). It seems to be a persistent problem - coexistence SELINIX and RDO Openstack (Havana,IceHouse,Juno). Anyway, to make any conclusions packstack has to be run with SELINUX set to permissive mode. In particular case (Antonio's install) analyzing corresponding logs after packstack completion will clearly show SELINUX related problems during packstack installation and may explain why internal interface DOWN state on neutron demo's router for demo_private sub-net has been set.
It's not quite clear from Antonio's answer when he succeeded with private network and router creation for tenant , before or after set SELINUX to permissive mode. That is my question to him.

edit flag offensive delete link more

Comments

I succeeded just AFTER set SELinux to permissive.

Antonio G. gravatar imageAntonio G. ( 2014-12-10 06:58:18 -0600 )edit

Please, upload text file

sudo cat /var/log/audit/audit.log | grep -i avc > avc.txt
DanIzack gravatar imageDanIzack ( 2014-12-10 07:26:30 -0600 )edit

No result for both commands:

[root@juno61-60-controller ~]# sudo cat /var/log/audit/audit.log | grep -i avc
[root@juno61-60-controller ~]# grep dnsmasq /var/log/audit/audit.log
Antonio G. gravatar imageAntonio G. ( 2014-12-10 08:00:13 -0600 )edit

If it's development box , please set in /etc/selinux/config

SELINUX=enforcing

system reboot and see what happens to dnsmasq now working fine with new private subnet

DanIzack gravatar imageDanIzack ( 2014-12-10 08:38:31 -0600 )edit
1

answered 2014-12-09 05:27:45 -0600

dbaxps gravatar image

updated 2014-12-11 08:24:58 -0600

UPDATE 2

 RDO Juno AIO install on CentOS 7 (KVM) with SELINUX enforced 
    has been done. During install there was just one AVC denial alert
    related with Nova , not with Neutron. After install I attempted to activate
    DHCP for demo_private network via dnsmasq and failed. AVC denial
    protocol was captured and uploaded to following location along with
    my actions.

View :- http://bderzhavets.blogspot.com/2014/...

UPDATE 1

You wrote : you were right, I forgot to set it to permissive through setenforce 0

Per official RDO instructions SELINUX should be set to permissive before running packstack

Dnsmasq makes an attempt set attribute on /var/log/neutron folder && search in

/var/lib/neutron
/var/log/neutron

directories , which gets blocked SELINUX no matter of current 755. Could you,please,
upload somewhere "grep dnsmasq /var/log/audit/audit.log".
As far AVC denial would come up it would be a fair.
I've installed Juno on CentOS 7 for MultiNode Deployment several times
and never had problems with demo_network created by packstack.
View also https://bugzilla.redhat.com/show_bug....

There is another way to test . I just guess :
Enable SELINUX  and your new private network which is fine right now,
will demonstrate dnsmasq access problems, regardless 755 will  be still in place.
Just  see Warnings in "service neutron-dhcp-agent status -l" on my working Network Node.

====================================================================================

[root@juno1 ~(keystone_admin)]# service neutron-dhcp-agent status
Redirecting to /bin/systemctl status  neutron-dhcp-agent.service
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: active (running) since Tue 2014-12-09 07:49:42 MSK; 6h ago
 Main PID: 3585 (neutron-dhcp-ag)
   CGroup: /system.slice/neutron-dhcp-agent.service
           ├─3585 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron...
           ├─5872 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap305cf...
           ├─5879 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap5532b...
           ├─5887 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap37cfc...
           ├─5889 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap2d26c...
           └─6015 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapec79a...

Dec 09 07:50:10 juno1.localdomain sudo[5943]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/... up
Dec 09 07:50:10 juno1.localdomain sudo[5949]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...faa
Dec 09 07:50:10 juno1.localdomain sudo[5956]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/... up
Dec 09 07:50:11 juno1.localdomain sudo[5973]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...bal
Dec 09 07:50:11 juno1.localdomain sudo[5987]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...-04
Dec 09 07:50:11 juno1.localdomain sudo[5993]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...ink
Dec 09 07:50:11 juno1.localdomain sudo[5999]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...-04
Dec 09 07:50:11 juno1.localdomain sudo[6003]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...-04
Dec 09 07:50:12 juno1.localdomain sudo[6008]: neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/...ace
Dec 09 07:50:12 juno1.localdomain python[5875]: SELinux is preventing /usr/sbin/dnsmasq from search...y .

                                                *****  Plugin catchall (100. confidence) suggests   **...
Dec ...
(more)
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-12-09 04:11:02 -0600

Seen: 953 times

Last updated: Dec 11 '14