Revision history [back]

click to hide/show revision 1
initial version

privsep not running when starting nova and/or neutron

Openstack (Pike) is running with 6 compute nodes and one controller. Centos 7 is installed on all servers.

Nova and Neutron are running fine on 3 compute nodes.

On the other three nodes, when I start Nova and Neutron services both of them start without error but privsep is not running and no error logged.

Thus, iptables is not configured and ressources can't be allocated on those physical servers. Nova.conf and neutron.conf are identical on all compute nodes

[root@node1 nova]# systemctl start openstack-nova-compute.service
[root@node1 nova]# systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:41:19 CEST; 13s ago
 Main PID: 3822829 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─3822829 /usr/bin/python2 /usr/bin/nova-compute

avr 24 10:41:15 node1 systemd[1]: Starting OpenStack Nova Compute Server...
avr 24 10:41:19 node1 systemd[1]: Started OpenStack Nova Compute Server.


[root@node1 nova]# ps -ef | grep nova
nova     3822829       1  6 10:41 ?        00:00:05 /usr/bin/python2 /usr/bin/nova-compute
root     3823029 3817406  0 10:42 pts/0    00:00:00 grep --color=auto nova


[root@node1 nova]# ps -ef | grep neutron
neutron  3824601       1 99 10:47 ?        00:00:01 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log


[root@node1 nova]# systemctl status -l neutron-linuxbridge-agent.service
● neutron-linuxbridge-agent.service - OpenStack Neutron Linux Bridge Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-linuxbridge-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:57:53 CEST; 2min 36s ago
  Process: 3824594 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS)
 Main PID: 3824601 (neutron-linuxbr)
   CGroup: /system.slice/neutron-linuxbridge-agent.service
           ├─3824601 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log
           ├─3824623 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
           └─3824624 /usr/bin/python2 /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:53 node1 systemd[1]: Starting OpenStack Neutron Linux Bridge Agent...
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-iptables = 1
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-ip6tables = 1
avr 24 10:57:53 node1 systemd[1]: Started OpenStack Neutron Linux Bridge Agent.
avr 24 10:57:54 node1 neutron-linuxbridge-agent[3824601]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
avr 24 10:57:55 node1 sudo[3824623]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: Starting rootwrap daemon main loop
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/iptables-save', u'-t', u'raw'] (filter match = iptables-save)
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/ebtables', u'--concurrent', u'-L'] (filter match = ebtables)

Nothing related to privsep in logs:

[root@node1 log]# cat nova/privsep-helper.log
[root@node1 log]#

Here are some amqp error messages:

[root@node1 nova]# grep -iv info /var/log/neutron/linuxbridge-agent.log
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 128, in _report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     True)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 92, in report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return method(context, 'report_state', **kwargs)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=self.retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     timeout=timeout, retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 555, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = self._waiter.wait(msg_id, timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in wait
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     message = self.waiters.get(msg_id, timeout=timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 335, in get
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     'to message ID %s' % msg_id)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent
2018-04-24 10:56:23.463 2088259 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.agent._common_agent.CommonAgentLoop._report_state' run outlasted interval by 30.01 sec

rabbitmq seems to be functionnal:

[root@openstack]# lsof -i :5672 | grep node1
beam.smp    14935 rabbitmq  248u  IPv6 2214296743      0t0  TCP openstack:amqp->node1:42251 (ESTABLISHED)
beam.smp    14935 rabbitmq  336u  IPv6 2214296746      0t0  TCP openstack:amqp->node1:42252 (ESTABLISHED)
beam.smp    14935 rabbitmq  379u  IPv6 2214296750      0t0  TCP openstack:amqp->node1:42265 (ESTABLISHED)
beam.smp    14935 rabbitmq  383u  IPv6 2214359591      0t0  TCP openstack:amqp->node1:42431 (ESTABLISHED)
beam.smp    14935 rabbitmq  384u  IPv6 2214359594      0t0  TCP openstack:amqp->node1:42432 (ESTABLISHED)
beam.smp    14935 rabbitmq  385u  IPv6 2214359596      0t0  TCP openstack:amqp->node1:42433 (ESTABLISHED)
beam.smp    14935 rabbitmq  390u  IPv6 2214359598      0t0  TCP openstack:amqp->node1:42434 (ESTABLISHED)
beam.smp    14935 rabbitmq  395u  IPv6 2214359601      0t0  TCP openstack:amqp->node1:42435 (ESTABLISHED)
beam.smp    14935 rabbitmq  397u  IPv6 2214359603      0t0  TCP openstack:amqp->node1:42436 (ESTABLISHED)
beam.smp    14935 rabbitmq  413u  IPv6 2214359606      0t0  TCP openstack:amqp->node1:42437 (ESTABLISHED)
beam.smp    14935 rabbitmq  417u  IPv6 2214359609      0t0  TCP openstack:amqp->node1:42438 (ESTABLISHED)
beam.smp    14935 rabbitmq  486u  IPv6 2214296759      0t0  TCP openstack:amqp->node1:42266 (ESTABLISHED)


[root@node1 nova]# lsof -i :5672 | grep openstack
nova-comp 3822829    nova    4u  IPv4 25228156      0t0  TCP node1:42251->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova    5u  IPv4 25354040      0t0  TCP node1:42252->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   21u  IPv4 25228167      0t0  TCP node1:42265->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   22u  IPv4 25584694      0t0  TCP node1:42266->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron    7u  IPv4 25307590      0t0  TCP node1:42431->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   11u  IPv4 25307591      0t0  TCP node1:42432->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   12u  IPv4 25349624      0t0  TCP node1:42433->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   13u  IPv4 25268077      0t0  TCP node1:42434->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   14u  IPv4 25307593      0t0  TCP node1:42435->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   15u  IPv4 25349626      0t0  TCP node1:42436->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   16u  IPv4 25594915      0t0  TCP node1:42437->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   17u  IPv4 25576890      0t0  TCP node1:42438->openstack:amqp (ESTABLISHED)

Versions installed:

[root@node1 nova]# rpm -qa | grep nova
openstack-nova-compute-16.0.3-2.el7.noarch
python-nova-16.0.3-2.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
python2-novaclient-9.1.1-1.el7.noarch

[root@node1 nova]# rpm -qa | grep neutron
python-neutron-lib-1.9.1-1.el7.noarch
python-neutron-11.0.2-3.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
python2-neutronclient-6.5.0-1.el7.noarch


[root@node1 nova]# rpm -qa | grep privsep
python2-oslo-privsep-1.22.1-1.el7.noarch
python-oslo-privsep-lang-1.22.1-1.el7.noarch


[root@openstack]# rpm -qa | grep openstack
python2-openstacksdk-0.9.17-1.el7.noarch
openstack-nova-scheduler-16.0.3-2.el7.noarch
openstack-nova-compute-16.0.3-2.el7.noarch
python2-django-openstack-auth-3.5.0-1.el7.noarch
openstack-nova-console-16.0.3-2.el7.noarch
openstack-neutron-11.0.2-3.el7.noarch
centos-release-openstack-pike-1-1.el7.x86_64
openstack-neutron-ml2-11.0.2-3.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
openstack-nova-conductor-16.0.3-2.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
openstack-cinder-11.1.0-1.el7.noarch
openstack-keystone-12.0.0-1.el7.noarch
openstack-nova-api-16.0.3-2.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
python2-openstackclient-3.12.0-1.el7.noarch
openstack-glance-15.0.0-2.el7.noarch
openstack-nova-placement-api-16.0.3-2.el7.noarch
openstack-dashboard-12.0.2-1.el7.noarch
openstack-nova-novncproxy-16.0.3-2.el7.noarch
openstack-utils-2017.1-1.el7.noarch
python-openstackclient-lang-3.12.0-1.el7.noarch

Node1 is listed by nova-manage on my openstack server:

[root@openstack]# nova-manage host list | grep node1
host           zone             
node1       nova

What could be wrong with privsep and/or nova/neutron ?

What about the messaging timeout ?

Thanks for your help.

privsep not running when starting nova and/or neutron

Openstack (Pike) is running with 6 compute nodes and one controller. Centos 7 is installed on all servers.

Nova and Neutron are running fine on 3 compute nodes.

On the other three nodes, when I start Nova and Neutron services both of them start without error but privsep is not running and no error logged.

Thus, iptables is not configured and ressources can't be allocated on those physical servers. Nova.conf and neutron.conf are identical on all compute nodes

[root@node1 nova]# systemctl start openstack-nova-compute.service
[root@node1 nova]# systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:41:19 CEST; 13s ago
 Main PID: 3822829 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─3822829 /usr/bin/python2 /usr/bin/nova-compute

avr 24 10:41:15 node1 systemd[1]: Starting OpenStack Nova Compute Server...
avr 24 10:41:19 node1 systemd[1]: Started OpenStack Nova Compute Server.


[root@node1 nova]# ps -ef | grep nova
nova     3822829       1  6 10:41 ?        00:00:05 /usr/bin/python2 /usr/bin/nova-compute
root     3823029 3817406  0 10:42 pts/0    00:00:00 grep --color=auto nova


[root@node1 nova]# ps -ef | grep neutron
neutron  3824601       1 99 10:47 ?        00:00:01 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log


[root@node1 nova]# systemctl status -l neutron-linuxbridge-agent.service
● neutron-linuxbridge-agent.service - OpenStack Neutron Linux Bridge Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-linuxbridge-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:57:53 CEST; 2min 36s ago
  Process: 3824594 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS)
 Main PID: 3824601 (neutron-linuxbr)
   CGroup: /system.slice/neutron-linuxbridge-agent.service
           ├─3824601 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log
           ├─3824623 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
           └─3824624 /usr/bin/python2 /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:53 node1 systemd[1]: Starting OpenStack Neutron Linux Bridge Agent...
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-iptables = 1
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-ip6tables = 1
avr 24 10:57:53 node1 systemd[1]: Started OpenStack Neutron Linux Bridge Agent.
avr 24 10:57:54 node1 neutron-linuxbridge-agent[3824601]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
avr 24 10:57:55 node1 sudo[3824623]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: Starting rootwrap daemon main loop
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/iptables-save', u'-t', u'raw'] (filter match = iptables-save)
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/ebtables', u'--concurrent', u'-L'] (filter match = ebtables)

Nothing related to privsep in logs:

[root@node1 log]# cat nova/privsep-helper.log
[root@node1 log]#

Here are some amqp error messages:

[root@node1 nova]# grep -iv info /var/log/neutron/linuxbridge-agent.log
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 128, in _report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     True)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 92, in report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return method(context, 'report_state', **kwargs)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=self.retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     timeout=timeout, retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 555, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = self._waiter.wait(msg_id, timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in wait
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     message = self.waiters.get(msg_id, timeout=timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 335, in get
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     'to message ID %s' % msg_id)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent
2018-04-24 10:56:23.463 2088259 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.agent._common_agent.CommonAgentLoop._report_state' run outlasted interval by 30.01 sec

rabbitmq seems to be functionnal:

[root@openstack]# lsof -i :5672 | grep node1
beam.smp    14935 rabbitmq  248u  IPv6 2214296743      0t0  TCP openstack:amqp->node1:42251 (ESTABLISHED)
beam.smp    14935 rabbitmq  336u  IPv6 2214296746      0t0  TCP openstack:amqp->node1:42252 (ESTABLISHED)
beam.smp    14935 rabbitmq  379u  IPv6 2214296750      0t0  TCP openstack:amqp->node1:42265 (ESTABLISHED)
beam.smp    14935 rabbitmq  383u  IPv6 2214359591      0t0  TCP openstack:amqp->node1:42431 (ESTABLISHED)
beam.smp    14935 rabbitmq  384u  IPv6 2214359594      0t0  TCP openstack:amqp->node1:42432 (ESTABLISHED)
beam.smp    14935 rabbitmq  385u  IPv6 2214359596      0t0  TCP openstack:amqp->node1:42433 (ESTABLISHED)
beam.smp    14935 rabbitmq  390u  IPv6 2214359598      0t0  TCP openstack:amqp->node1:42434 (ESTABLISHED)
beam.smp    14935 rabbitmq  395u  IPv6 2214359601      0t0  TCP openstack:amqp->node1:42435 (ESTABLISHED)
beam.smp    14935 rabbitmq  397u  IPv6 2214359603      0t0  TCP openstack:amqp->node1:42436 (ESTABLISHED)
beam.smp    14935 rabbitmq  413u  IPv6 2214359606      0t0  TCP openstack:amqp->node1:42437 (ESTABLISHED)
beam.smp    14935 rabbitmq  417u  IPv6 2214359609      0t0  TCP openstack:amqp->node1:42438 (ESTABLISHED)
beam.smp    14935 rabbitmq  486u  IPv6 2214296759      0t0  TCP openstack:amqp->node1:42266 (ESTABLISHED)


[root@node1 nova]# lsof -i :5672 | grep openstack
nova-comp 3822829    nova    4u  IPv4 25228156      0t0  TCP node1:42251->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova    5u  IPv4 25354040      0t0  TCP node1:42252->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   21u  IPv4 25228167      0t0  TCP node1:42265->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   22u  IPv4 25584694      0t0  TCP node1:42266->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron    7u  IPv4 25307590      0t0  TCP node1:42431->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   11u  IPv4 25307591      0t0  TCP node1:42432->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   12u  IPv4 25349624      0t0  TCP node1:42433->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   13u  IPv4 25268077      0t0  TCP node1:42434->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   14u  IPv4 25307593      0t0  TCP node1:42435->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   15u  IPv4 25349626      0t0  TCP node1:42436->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   16u  IPv4 25594915      0t0  TCP node1:42437->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   17u  IPv4 25576890      0t0  TCP node1:42438->openstack:amqp (ESTABLISHED)

Versions installed:

[root@node1 nova]# rpm -qa | grep nova
openstack-nova-compute-16.0.3-2.el7.noarch
python-nova-16.0.3-2.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
python2-novaclient-9.1.1-1.el7.noarch

[root@node1 nova]# rpm -qa | grep neutron
python-neutron-lib-1.9.1-1.el7.noarch
python-neutron-11.0.2-3.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
python2-neutronclient-6.5.0-1.el7.noarch


[root@node1 nova]# rpm -qa | grep privsep
python2-oslo-privsep-1.22.1-1.el7.noarch
python-oslo-privsep-lang-1.22.1-1.el7.noarch


[root@openstack]# rpm -qa | grep openstack
python2-openstacksdk-0.9.17-1.el7.noarch
openstack-nova-scheduler-16.0.3-2.el7.noarch
openstack-nova-compute-16.0.3-2.el7.noarch
python2-django-openstack-auth-3.5.0-1.el7.noarch
openstack-nova-console-16.0.3-2.el7.noarch
openstack-neutron-11.0.2-3.el7.noarch
centos-release-openstack-pike-1-1.el7.x86_64
openstack-neutron-ml2-11.0.2-3.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
openstack-nova-conductor-16.0.3-2.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
openstack-cinder-11.1.0-1.el7.noarch
openstack-keystone-12.0.0-1.el7.noarch
openstack-nova-api-16.0.3-2.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
python2-openstackclient-3.12.0-1.el7.noarch
openstack-glance-15.0.0-2.el7.noarch
openstack-nova-placement-api-16.0.3-2.el7.noarch
openstack-dashboard-12.0.2-1.el7.noarch
openstack-nova-novncproxy-16.0.3-2.el7.noarch
openstack-utils-2017.1-1.el7.noarch
python-openstackclient-lang-3.12.0-1.el7.noarch

Node1 is listed by nova-manage on my openstack server:

[root@openstack]# nova-manage host list | grep node1
host           zone             
node1       nova

What could be wrong with privsep and/or nova/neutron ?

What about the messaging timeout ?

Thanks for your help.

privsep not running when starting nova and/or neutron

Openstack (Pike) is running with 6 compute nodes and one controller. Centos 7 is installed on all servers.

Nova and Neutron are running fine on 3 compute nodes.

On the other three nodes, when I start Nova and Neutron services both of them start without error but privsep is not running and no error logged.

Thus, iptables is not configured and ressources can't be allocated on those physical servers. Nova.conf and neutron.conf are identical on all compute nodes

[root@node1 nova]# systemctl start openstack-nova-compute.service
[root@node1 nova]# systemctl status -l openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:41:19 CEST; 13s ago
 Main PID: 3822829 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─3822829 /usr/bin/python2 /usr/bin/nova-compute

avr 24 10:41:15 node1 systemd[1]: Starting OpenStack Nova Compute Server...
avr 24 10:41:19 node1 systemd[1]: Started OpenStack Nova Compute Server.


[root@node1 nova]# ps -ef | grep nova
nova     3822829       1  6 10:41 ?        00:00:05 /usr/bin/python2 /usr/bin/nova-compute
root     3823029 3817406  0 10:42 pts/0    00:00:00 grep --color=auto nova


[root@node1 nova]# ps -ef | grep neutron
neutron  3824601       1 99 10:47 ?        00:00:01 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log


[root@node1 nova]# systemctl status -l neutron-linuxbridge-agent.service
● neutron-linuxbridge-agent.service - OpenStack Neutron Linux Bridge Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-linuxbridge-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since mar 2018-04-24 10:57:53 CEST; 2min 36s ago
  Process: 3824594 ExecStartPre=/usr/bin/neutron-enable-bridge-firewall.sh (code=exited, status=0/SUCCESS)
 Main PID: 3824601 (neutron-linuxbr)
   CGroup: /system.slice/neutron-linuxbridge-agent.service
           ├─3824601 /usr/bin/python2 /usr/bin/neutron-linuxbridge-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/linuxbridge_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-linuxbridge-agent --log-file /var/log/neutron/linuxbridge-agent.log
           ├─3824623 sudo neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
           └─3824624 /usr/bin/python2 /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:53 node1 systemd[1]: Starting OpenStack Neutron Linux Bridge Agent...
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-iptables = 1
avr 24 10:57:53 node1 neutron-enable-bridge-firewall.sh[3824594]: net.bridge.bridge-nf-call-ip6tables = 1
avr 24 10:57:53 node1 systemd[1]: Started OpenStack Neutron Linux Bridge Agent.
avr 24 10:57:54 node1 neutron-linuxbridge-agent[3824601]: Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
avr 24 10:57:55 node1 sudo[3824623]:  neutron : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: Starting rootwrap daemon main loop
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/iptables-save', u'-t', u'raw'] (filter match = iptables-save)
avr 24 10:57:55 node1 neutron-rootwrap-daemon[3824624]: (root > root) Executing ['/sbin/ebtables', u'--concurrent', u'-L'] (filter match = ebtables)

Nothing related to privsep in logs:

[root@node1 log]# cat nova/privsep-helper.log
[root@node1 log]#

Here are some amqp error messages:

[root@node1 nova]# grep -iv info /var/log/neutron/linuxbridge-agent.log
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 128, in _report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     True)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 92, in report_state
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return method(context, 'report_state', **kwargs)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=self.retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     timeout=timeout, retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     retry=retry)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 555, in _send
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = self._waiter.wait(msg_id, timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in wait
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     message = self.waiters.get(msg_id, timeout=timeout)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 335, in get
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent     'to message ID %s' % msg_id)
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent MessagingTimeout: Timed out waiting for a reply to message ID b42e605ac1c04acf981f05365917b7b5
2018-04-24 10:56:23.462 2088259 ERROR neutron.plugins.ml2.drivers.agent._common_agent
2018-04-24 10:56:23.463 2088259 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.agent._common_agent.CommonAgentLoop._report_state' run outlasted interval by 30.01 sec

rabbitmq seems to be functionnal:

[root@openstack]# lsof -i :5672 | grep node1
beam.smp    14935 rabbitmq  248u  IPv6 2214296743      0t0  TCP openstack:amqp->node1:42251 (ESTABLISHED)
beam.smp    14935 rabbitmq  336u  IPv6 2214296746      0t0  TCP openstack:amqp->node1:42252 (ESTABLISHED)
beam.smp    14935 rabbitmq  379u  IPv6 2214296750      0t0  TCP openstack:amqp->node1:42265 (ESTABLISHED)
beam.smp    14935 rabbitmq  383u  IPv6 2214359591      0t0  TCP openstack:amqp->node1:42431 (ESTABLISHED)
beam.smp    14935 rabbitmq  384u  IPv6 2214359594      0t0  TCP openstack:amqp->node1:42432 (ESTABLISHED)
beam.smp    14935 rabbitmq  385u  IPv6 2214359596      0t0  TCP openstack:amqp->node1:42433 (ESTABLISHED)
beam.smp    14935 rabbitmq  390u  IPv6 2214359598      0t0  TCP openstack:amqp->node1:42434 (ESTABLISHED)
beam.smp    14935 rabbitmq  395u  IPv6 2214359601      0t0  TCP openstack:amqp->node1:42435 (ESTABLISHED)
beam.smp    14935 rabbitmq  397u  IPv6 2214359603      0t0  TCP openstack:amqp->node1:42436 (ESTABLISHED)
beam.smp    14935 rabbitmq  413u  IPv6 2214359606      0t0  TCP openstack:amqp->node1:42437 (ESTABLISHED)
beam.smp    14935 rabbitmq  417u  IPv6 2214359609      0t0  TCP openstack:amqp->node1:42438 (ESTABLISHED)
beam.smp    14935 rabbitmq  486u  IPv6 2214296759      0t0  TCP openstack:amqp->node1:42266 (ESTABLISHED)


[root@node1 nova]# lsof -i :5672 | grep openstack
nova-comp 3822829    nova    4u  IPv4 25228156      0t0  TCP node1:42251->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova    5u  IPv4 25354040      0t0  TCP node1:42252->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   21u  IPv4 25228167      0t0  TCP node1:42265->openstack:amqp (ESTABLISHED)
nova-comp 3822829    nova   22u  IPv4 25584694      0t0  TCP node1:42266->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron    7u  IPv4 25307590      0t0  TCP node1:42431->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   11u  IPv4 25307591      0t0  TCP node1:42432->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   12u  IPv4 25349624      0t0  TCP node1:42433->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   13u  IPv4 25268077      0t0  TCP node1:42434->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   14u  IPv4 25307593      0t0  TCP node1:42435->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   15u  IPv4 25349626      0t0  TCP node1:42436->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   16u  IPv4 25594915      0t0  TCP node1:42437->openstack:amqp (ESTABLISHED)
neutron-l 3824601 neutron   17u  IPv4 25576890      0t0  TCP node1:42438->openstack:amqp (ESTABLISHED)

Versions installed:

[root@node1 nova]# rpm -qa | grep nova
openstack-nova-compute-16.0.3-2.el7.noarch
python-nova-16.0.3-2.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
python2-novaclient-9.1.1-1.el7.noarch

[root@node1 nova]# rpm -qa | grep neutron
python-neutron-lib-1.9.1-1.el7.noarch
python-neutron-11.0.2-3.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
python2-neutronclient-6.5.0-1.el7.noarch


[root@node1 nova]# rpm -qa | grep privsep
python2-oslo-privsep-1.22.1-1.el7.noarch
python-oslo-privsep-lang-1.22.1-1.el7.noarch


[root@openstack]# rpm -qa | grep openstack
python2-openstacksdk-0.9.17-1.el7.noarch
openstack-nova-scheduler-16.0.3-2.el7.noarch
openstack-nova-compute-16.0.3-2.el7.noarch
python2-django-openstack-auth-3.5.0-1.el7.noarch
openstack-nova-console-16.0.3-2.el7.noarch
openstack-neutron-11.0.2-3.el7.noarch
centos-release-openstack-pike-1-1.el7.x86_64
openstack-neutron-ml2-11.0.2-3.el7.noarch
openstack-nova-common-16.0.3-2.el7.noarch
openstack-nova-conductor-16.0.3-2.el7.noarch
openstack-neutron-common-11.0.2-3.el7.noarch
openstack-cinder-11.1.0-1.el7.noarch
openstack-keystone-12.0.0-1.el7.noarch
openstack-nova-api-16.0.3-2.el7.noarch
openstack-neutron-linuxbridge-11.0.2-3.el7.noarch
python2-openstackclient-3.12.0-1.el7.noarch
openstack-glance-15.0.0-2.el7.noarch
openstack-nova-placement-api-16.0.3-2.el7.noarch
openstack-dashboard-12.0.2-1.el7.noarch
openstack-nova-novncproxy-16.0.3-2.el7.noarch
openstack-utils-2017.1-1.el7.noarch
python-openstackclient-lang-3.12.0-1.el7.noarch

Node1 is listed by nova-manage on my openstack server:

[root@openstack]# nova-manage host list | grep node1
host           zone             
node1       nova

What could be wrong with privsep and/or nova/neutron ?

What about the messaging timeout ?

Thanks for your help.