Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

httpd.service fails to start in an HA config

hi, I am creating a Stein sandbox environment from rpms and am running stuck configuring it for HA (3 controllers).
I have a few issues I am sure. For now I am looking at one where the httpd.service is failing on controller-1.
I have the VIP set up and controller-1 has the VIP.
I must be running up against a port conflict? Am I required to use a network namespace for haproxy to run in?
Any advice in this area would be appreciated! (I can post more config files if you need thx!)

Here is the httpd status:

systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: failed (Result: exit-code) since Wed 2020-01-08 23:35:34 UTC; 3min 10s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 7946 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 7818 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
  Process: 2928 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 1501 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 7818 (code=exited, status=1/FAILURE)

Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address [::]:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address 0.0.0.0:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: no listening sockets available, shutting down
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: AH00015: Unable to open logs
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 08 23:35:34 stein-controller-1.dialogic.com kill[7946]: kill: cannot find process ""
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: Failed to start The Apache HTTP Server.
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: Unit httpd.service entered failed state.
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service failed.

Here is the pcsd status:

pcs status
Cluster name: stein_ha_cluster
Stack: corosync
Current DC: stein-controller-3 (version 1.1.20-5.el7_7.1-3c4c782f70) - partition with quorum
Last updated: Wed Jan  8 23:04:45 2020
Last change: Tue Dec 31 20:54:41 2019 by root via cibadmin on stein-controller-1

3 nodes configured
4 resources configured

Online: [ stein-controller-1 stein-controller-2 stein-controller-3 ]

Full list of resources:

 vip    (ocf::heartbeat:IPaddr2):       Started stein-controller-1
 Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ stein-controller-1 ]
     Stopped: [ stein-controller-2 stein-controller-3 ]

Failed Resource Actions:
* lb-haproxy_start_0 on stein-controller-2 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 22:50:16 2020', queued=0ms, exec=2444ms
* lb-haproxy_start_0 on stein-controller-3 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 17:51:53 2020', queued=0ms, exec=2415ms

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

And here is the VIP IP address up:

ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:85:23:4e brd ff:ff:ff:ff:ff:ff
    inet 10.27.240.41/24 brd 10.27.240.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.27.240.226/24 brd 10.27.240.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe85:234e/64 scope link
       valid_lft forever preferred_lft forever

Here is the haproxy.cfg file

 global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  user  haproxy

defaults
  log  global
  maxconn  4000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen dashboard_cluster
  bind 10.27.240.226:443
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:443 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:443 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:443 check inter 2000 rise 2 fall 5

 listen galera_cluster
  bind 10.27.240.226:3306
  balance  source
  option  mysql-check
  server stein-controller-1 10.27.240.41:3306 check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:3306 backup check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:3306 backup check port 9200 inter 2000 rise 2 fall 5

 listen glance_api_cluster
  bind 10.27.240.226:9292
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9292 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9292 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9292 check inter 2000 rise 2 fall 5

 listen glance_registry_cluster
  bind 10.27.240.226:9191
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:9191 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9191 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9191 check inter 2000 rise 2 fall 5

 listen keystone_admin_cluster
  bind 10.27.240.226:35357
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:35357 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:35357 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:35357 check inter 2000 rise 2 fall 5

 listen keystone_public_internal_cluster
  bind 10.27.240.226:5000
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:5000 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:5000 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:5000 check inter 2000 rise 2 fall 5

 listen nova_ec2_api_cluster
  bind 10.27.240.226:8773
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8773 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8773 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8773 check inter 2000 rise 2 fall 5

 listen nova_compute_api_cluster
  bind 10.27.240.226:8774
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8774 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8774 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8774 check inter 2000 rise 2 fall 5

 listen nova_metadata_api_cluster
  bind 10.27.240.226:8775
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8775 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8775 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8775 check inter 2000 rise 2 fall 5

 listen cinder_api_cluster
  bind 10.27.240.226:8776
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8776 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8776 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8776 check inter 2000 rise 2 fall 5

 listen ceilometer_api_cluster
  bind 10.27.240.226:8777
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8777 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8777 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8777 check inter 2000 rise 2 fall 5

 listen nova_vncproxy_cluster
  bind 10.27.240.226:6080
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:6080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:6080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:6080 check inter 2000 rise 2 fall 5

 listen neutron_api_cluster
  bind 10.27.240.226:9696
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9696 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9696 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9696 check inter 2000 rise 2 fall 5

 listen swift_proxy_cluster
  bind 10.27.240.226:8080
  balance  source
  option  tcplog
  option  tcpka
  server stein-controller-1 10.27.240.41:8080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8080 check inter 2000 rise 2 fall 5

listen stats 10.27.240.41:8999
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:ADMIN_PASS

Here is the httpd.conf file:

ServerRoot "/etc/httpd"
Listen 80
Include conf.modules.d/*.conf
User apache
Group apache
ServerAdmin root@localhost
ServerName stein-controller-1.dialogic.com
<Directory />
    AllowOverride none
    Require all denied
</Directory>
DocumentRoot "/var/www/html"
<Directory "/var/www">
    AllowOverride None
    Require all granted
</Directory>
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
<Files ".ht*">
    Require all denied
</Files>
ErrorLog "logs/error_log"
LogLevel warn
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>
<IfModule mime_module>
    TypesConfig /etc/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
    MIMEMagicFile conf/magic
</IfModule>
EnableSendfile on
IncludeOptional conf.d/*.conf

httpd.service fails to start in an HA config

hi, I am creating a Stein sandbox environment from rpms and am running stuck configuring it for HA (3 controllers).
I have a few issues I am sure. For now I am looking at one where the httpd.service is failing on controller-1.
I have the VIP set up and controller-1 has the VIP.
I must be running up against a port conflict? Am I required to use a network namespace for haproxy to run in?
Any advice in this area would be appreciated! (I can post more config files if you need thx!)

Here is a netstat:

  netstat -anp | grep "5000"
  tcp      0    0 10.20.247.226:5000    0.0.0.0:*               LISTEN      1461/haproxy
  tcp      0    0 10.20.247.41:50000    10.20.247.41:3306       TIME_WAIT   -

Here is the httpd status:

systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: failed (Result: exit-code) since Wed 2020-01-08 23:35:34 UTC; 3min 10s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 7946 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 7818 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
  Process: 2928 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 1501 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 7818 (code=exited, status=1/FAILURE)

Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address [::]:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address 0.0.0.0:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: no listening sockets available, shutting down
Jan 08 23:35:34 stein-controller-1.dialogic.com httpd[7818]: AH00015: Unable to open logs
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 08 23:35:34 stein-controller-1.dialogic.com kill[7946]: kill: cannot find process ""
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: Failed to start The Apache HTTP Server.
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: Unit httpd.service entered failed state.
Jan 08 23:35:34 stein-controller-1.dialogic.com systemd[1]: httpd.service failed.

Here is the pcsd status:

pcs status
Cluster name: stein_ha_cluster
Stack: corosync
Current DC: stein-controller-3 (version 1.1.20-5.el7_7.1-3c4c782f70) - partition with quorum
Last updated: Wed Jan  8 23:04:45 2020
Last change: Tue Dec 31 20:54:41 2019 by root via cibadmin on stein-controller-1

3 nodes configured
4 resources configured

Online: [ stein-controller-1 stein-controller-2 stein-controller-3 ]

Full list of resources:

 vip    (ocf::heartbeat:IPaddr2):       Started stein-controller-1
 Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ stein-controller-1 ]
     Stopped: [ stein-controller-2 stein-controller-3 ]

Failed Resource Actions:
* lb-haproxy_start_0 on stein-controller-2 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 22:50:16 2020', queued=0ms, exec=2444ms
* lb-haproxy_start_0 on stein-controller-3 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 17:51:53 2020', queued=0ms, exec=2415ms

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

And here is the VIP IP address up:

ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:85:23:4e brd ff:ff:ff:ff:ff:ff
    inet 10.27.240.41/24 brd 10.27.240.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.27.240.226/24 brd 10.27.240.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe85:234e/64 scope link
       valid_lft forever preferred_lft forever

Here is the haproxy.cfg file

 global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  user  haproxy

defaults
  log  global
  maxconn  4000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen dashboard_cluster
  bind 10.27.240.226:443
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:443 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:443 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:443 check inter 2000 rise 2 fall 5

 listen galera_cluster
  bind 10.27.240.226:3306
  balance  source
  option  mysql-check
  server stein-controller-1 10.27.240.41:3306 check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:3306 backup check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:3306 backup check port 9200 inter 2000 rise 2 fall 5

 listen glance_api_cluster
  bind 10.27.240.226:9292
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9292 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9292 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9292 check inter 2000 rise 2 fall 5

 listen glance_registry_cluster
  bind 10.27.240.226:9191
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:9191 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9191 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9191 check inter 2000 rise 2 fall 5

 listen keystone_admin_cluster
  bind 10.27.240.226:35357
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:35357 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:35357 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:35357 check inter 2000 rise 2 fall 5

 listen keystone_public_internal_cluster
  bind 10.27.240.226:5000
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:5000 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:5000 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:5000 check inter 2000 rise 2 fall 5

 listen nova_ec2_api_cluster
  bind 10.27.240.226:8773
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8773 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8773 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8773 check inter 2000 rise 2 fall 5

 listen nova_compute_api_cluster
  bind 10.27.240.226:8774
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8774 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8774 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8774 check inter 2000 rise 2 fall 5

 listen nova_metadata_api_cluster
  bind 10.27.240.226:8775
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8775 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8775 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8775 check inter 2000 rise 2 fall 5

 listen cinder_api_cluster
  bind 10.27.240.226:8776
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8776 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8776 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8776 check inter 2000 rise 2 fall 5

 listen ceilometer_api_cluster
  bind 10.27.240.226:8777
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8777 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8777 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8777 check inter 2000 rise 2 fall 5

 listen nova_vncproxy_cluster
  bind 10.27.240.226:6080
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:6080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:6080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:6080 check inter 2000 rise 2 fall 5

 listen neutron_api_cluster
  bind 10.27.240.226:9696
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9696 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9696 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9696 check inter 2000 rise 2 fall 5

 listen swift_proxy_cluster
  bind 10.27.240.226:8080
  balance  source
  option  tcplog
  option  tcpka
  server stein-controller-1 10.27.240.41:8080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8080 check inter 2000 rise 2 fall 5

listen stats 10.27.240.41:8999
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:ADMIN_PASS

Here is the httpd.conf file:

ServerRoot "/etc/httpd"
Listen 80
Include conf.modules.d/*.conf
User apache
Group apache
ServerAdmin root@localhost
ServerName stein-controller-1.dialogic.com
<Directory />
    AllowOverride none
    Require all denied
</Directory>
DocumentRoot "/var/www/html"
<Directory "/var/www">
    AllowOverride None
    Require all granted
</Directory>
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
<Files ".ht*">
    Require all denied
</Files>
ErrorLog "logs/error_log"
LogLevel warn
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>
<IfModule mime_module>
    TypesConfig /etc/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
    MIMEMagicFile conf/magic
</IfModule>
EnableSendfile on
IncludeOptional conf.d/*.conf

httpd.service fails to start in an HA config

hi, I am creating a Stein sandbox environment from rpms and am running stuck configuring it for HA (3 controllers).
I have a few issues I am sure. For now I am looking at one where the httpd.service is failing on controller-1.
I have the VIP set up and controller-1 has the VIP.
I must be running up against a port conflict? Am I required to use a network namespace for haproxy to run in?
Any advice in this area would be appreciated! (I can post more config files if you need thx!)

Here is a netstat:

  netstat -anp | grep "5000"
  tcp      0    0 10.20.247.226:5000 10.27.240.226:5000    0.0.0.0:*               LISTEN      1461/haproxy
  tcp      0    0 10.20.247.41:50000    10.20.247.41:3306 10.27.240.41:50000    10.27.240.41:3306       TIME_WAIT   -

Here is the httpd status:

systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: failed (Result: exit-code) since Wed 2020-01-08 23:35:34 UTC; 3min 10s ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 7946 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 7818 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
  Process: 2928 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py compress --force -v0 (code=exited, status=0/SUCCESS)
  Process: 1501 ExecStartPre=/usr/bin/python2 /usr/share/openstack-dashboard/manage.py collectstatic --noinput --clear -v0 (code=exited, status=0/SUCCESS)
 Main PID: 7818 (code=exited, status=1/FAILURE)

Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address [::]:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com httpd[7818]: (98)Address already in use: AH00073: make_sock: unable to listen for connections on address 0.0.0.0:5000
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com httpd[7818]: no listening sockets available, shutting down
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com httpd[7818]: AH00015: Unable to open logs
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com kill[7946]: kill: cannot find process ""
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com systemd[1]: httpd.service: control process exited, code=exited status=1
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com systemd[1]: Failed to start The Apache HTTP Server.
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com systemd[1]: Unit httpd.service entered failed state.
Jan 08 23:35:34 stein-controller-1.dialogic.com stein-controller-1.mydom.com systemd[1]: httpd.service failed.

Here is the pcsd status:

pcs status
Cluster name: stein_ha_cluster
Stack: corosync
Current DC: stein-controller-3 (version 1.1.20-5.el7_7.1-3c4c782f70) - partition with quorum
Last updated: Wed Jan  8 23:04:45 2020
Last change: Tue Dec 31 20:54:41 2019 by root via cibadmin on stein-controller-1

3 nodes configured
4 resources configured

Online: [ stein-controller-1 stein-controller-2 stein-controller-3 ]

Full list of resources:

 vip    (ocf::heartbeat:IPaddr2):       Started stein-controller-1
 Clone Set: lb-haproxy-clone [lb-haproxy]
     Started: [ stein-controller-1 ]
     Stopped: [ stein-controller-2 stein-controller-3 ]

Failed Resource Actions:
* lb-haproxy_start_0 on stein-controller-2 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 22:50:16 2020', queued=0ms, exec=2444ms
* lb-haproxy_start_0 on stein-controller-3 'not running' (7): call=13, status=complete, exitreason='',
    last-rc-change='Wed Jan  8 17:51:53 2020', queued=0ms, exec=2415ms

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

And here is the VIP IP address up:

ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:85:23:4e brd ff:ff:ff:ff:ff:ff
    inet 10.27.240.41/24 brd 10.27.240.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.27.240.226/24 brd 10.27.240.255 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe85:234e/64 scope link
       valid_lft forever preferred_lft forever

Here is the haproxy.cfg file

 global
  chroot  /var/lib/haproxy
  daemon
  group  haproxy
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  user  haproxy

defaults
  log  global
  maxconn  4000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen dashboard_cluster
  bind 10.27.240.226:443
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:443 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:443 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:443 check inter 2000 rise 2 fall 5

 listen galera_cluster
  bind 10.27.240.226:3306
  balance  source
  option  mysql-check
  server stein-controller-1 10.27.240.41:3306 check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:3306 backup check port 9200 inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:3306 backup check port 9200 inter 2000 rise 2 fall 5

 listen glance_api_cluster
  bind 10.27.240.226:9292
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9292 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9292 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9292 check inter 2000 rise 2 fall 5

 listen glance_registry_cluster
  bind 10.27.240.226:9191
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:9191 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9191 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9191 check inter 2000 rise 2 fall 5

 listen keystone_admin_cluster
  bind 10.27.240.226:35357
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:35357 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:35357 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:35357 check inter 2000 rise 2 fall 5

 listen keystone_public_internal_cluster
  bind 10.27.240.226:5000
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:5000 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:5000 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:5000 check inter 2000 rise 2 fall 5

 listen nova_ec2_api_cluster
  bind 10.27.240.226:8773
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8773 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8773 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8773 check inter 2000 rise 2 fall 5

 listen nova_compute_api_cluster
  bind 10.27.240.226:8774
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8774 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8774 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8774 check inter 2000 rise 2 fall 5

 listen nova_metadata_api_cluster
  bind 10.27.240.226:8775
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8775 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8775 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8775 check inter 2000 rise 2 fall 5

 listen cinder_api_cluster
  bind 10.27.240.226:8776
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:8776 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8776 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8776 check inter 2000 rise 2 fall 5

 listen ceilometer_api_cluster
  bind 10.27.240.226:8777
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:8777 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8777 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8777 check inter 2000 rise 2 fall 5

 listen nova_vncproxy_cluster
  bind 10.27.240.226:6080
  balance  source
  option  tcpka
  option  tcplog
  server stein-controller-1 10.27.240.41:6080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:6080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:6080 check inter 2000 rise 2 fall 5

 listen neutron_api_cluster
  bind 10.27.240.226:9696
  balance  source
  option  tcpka
  option  httpchk
  option  tcplog
  server stein-controller-1 10.27.240.41:9696 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:9696 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:9696 check inter 2000 rise 2 fall 5

 listen swift_proxy_cluster
  bind 10.27.240.226:8080
  balance  source
  option  tcplog
  option  tcpka
  server stein-controller-1 10.27.240.41:8080 check inter 2000 rise 2 fall 5
  server stein-controller-2 10.27.240.40:8080 check inter 2000 rise 2 fall 5
  server stein-controller-3 10.27.240.20:8080 check inter 2000 rise 2 fall 5

listen stats 10.27.240.41:8999
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:ADMIN_PASS

Here is the httpd.conf file:

ServerRoot "/etc/httpd"
Listen 80
Include conf.modules.d/*.conf
User apache
Group apache
ServerAdmin root@localhost
ServerName stein-controller-1.dialogic.com
stein-controller-1.mydom.com
<Directory />
    AllowOverride none
    Require all denied
</Directory>
DocumentRoot "/var/www/html"
<Directory "/var/www">
    AllowOverride None
    Require all granted
</Directory>
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
<Files ".ht*">
    Require all denied
</Files>
ErrorLog "logs/error_log"
LogLevel warn
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>
<IfModule mime_module>
    TypesConfig /etc/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
    MIMEMagicFile conf/magic
</IfModule>
EnableSendfile on
IncludeOptional conf.d/*.conf