Keystone socket error "[Errno 98] Address already in use"

asked 2015-05-16 21:48:41 -0500

84 ●20 ●22 ●28 https://in.linkedin.co...

updated 2015-05-17 00:37:45 -0500

Hi,

I couldn't able to configure keystone on Killo, since I am getting socket error. Please find the below configuration and error details and help me to fix this issue..

Doc link: http://docs.openstack.org/kilo/instal...

==> /var/log/keystone/keystone.log <==
2015-05-17 07:54:38.106 3116 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/server/eventlet.py", line 54, in launch_with
2015-05-17 07:54:38.106 3116 TRACE keystone     self.server.listen()
2015-05-17 07:54:38.106 3116 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/common/environment/eventlet_server.py", line 105, in listen
2015-05-17 07:54:38.106 3116 TRACE keystone     backlog=backlog)
2015-05-17 07:54:38.106 3116 TRACE keystone   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2015-05-17 07:54:38.106 3116 TRACE keystone     sock.bind(addr)
2015-05-17 07:54:38.106 3116 TRACE keystone   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2015-05-17 07:54:38.106 3116 TRACE keystone     return getattr(self._sock,name)(*args)
2015-05-17 07:54:38.106 3116 TRACE keystone error: [Errno 98] Address already in use
2015-05-17 07:54:38.106 3116 TRACE keystone

[root@controler ~]# keystone-all --config-file /etc/keystone/keystone.conf
2015-05-17 08:00:59.310 3212 CRITICAL keystone [-] ConfigFileNotFound: An unexpected error prevented the server from fulfilling your request.
2015-05-17 08:00:59.310 3212 TRACE keystone Traceback (most recent call last):
2015-05-17 08:00:59.310 3212 TRACE keystone   File "/usr/bin/keystone-all", line 36, in <module>
2015-05-17 08:00:59.310 3212 TRACE keystone     eventlet_server.run(possible_topdir)
2015-05-17 08:00:59.310 3212 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/server/eventlet.py", line 135, in run
2015-05-17 08:00:59.310 3212 TRACE keystone     paste_config = config.find_paste_config()
2015-05-17 08:00:59.310 3212 TRACE keystone   File "/usr/lib/python2.7/site-packages/keystone/config.py", line 91, in find_paste_config
2015-05-17 08:00:59.310 3212 TRACE keystone     raise exception.ConfigFileNotFound(config_file=paste_config_value)
2015-05-17 08:00:59.310 3212 TRACE keystone ConfigFileNotFound: An unexpected error prevented the server from fulfilling your request.
2015-05-17 08:00:59.310 3212 TRACE keystone

[root@controler ~]# ps -ef | grep -i keystone
keystone  2759  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2760  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2761  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2762  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2763  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2764  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2765  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2766  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2767  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND
keystone  2768  1209  0 08:21 ?        00:00:00 (wsgi:keystone- -DFOREGROUND


/etc/keystone/keystone.conf
[DEFAULT]
admin_token = 8f5b05a867711c01afc1
log_dir = /var/log/keystone
verbose = True
[token ...

(more)

edit retag flag offensive close merge delete

add a comment

5 answers

Sort by » oldest newest most voted

answered 2015-11-24 07:53:28 -0500

Luigi

11 ●9 ●11 ●14 http://it.linkedin.com...

Have you considered that from Kilo release the keystone service is disabled ?

In Kilo and Liberty releases, the keystone project deprecates eventlet in favor of a separate web server with WSGI extensions. This guide uses the Apache HTTP server with mod_wsgi to serve Identity service requests on port 5000 and 35357. By default, the keystone service still listens on ports 5000 and 35357. Therefore, this guide disables the keystone service. The keystone project plans to remove eventlet support in Mitaka. Source

edit flag offensive delete link

add a comment

answered 2015-05-17 00:51:47 -0500

dbaxps-Second-Incarnation

106 ●1 ●4 ●7 http://bderzhavets.blo...

updated 2015-05-17 01:33:02 -0500

Attempt source credentials && run again :-

su -s /bin/sh -c "keystone-manage db_sync" keystone
service keystone restart

Looks like first instance is already running, you may also try identify process bound to port && kill it and try restart.

kill -9 `fuser -n tcp 5000`
kill -9 `fuser -n tcp 35357`

On working RDO Kilo Controller :-

[root@ip-192-169-142-127 ~(keystone_admin)]# systemctl | grep keystone
openstack-keystone.service                                                          loaded active running   OpenStack Identity Service (code-named Keystone)
[root@ip-192-169-142-127 ~(keystone_admin)]# netstat -lntp | grep 35357
tcp        0      0 0.0.0.0:35357           0.0.0.0:*               LISTEN      996/python          
[root@ip-192-169-142-127 ~(keystone_admin)]# ps -ef | 996
bash: 996: command not found...
[root@ip-192-169-142-127 ~(keystone_admin)]# ps -ef | grep 996
keystone   996     1  0 09:12 ?        00:00:05 /usr/bin/python /usr/bin/keystone-all
keystone  3975   996  0 09:13 ?        00:00:01 /usr/bin/python /usr/bin/keystone-all
keystone  3976   996  0 09:13 ?        00:00:02 /usr/bin/python /usr/bin/keystone-all
keystone  3977   996  0 09:13 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all
keystone  3978   996  0 09:13 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all
root      9496  5073  0 09:30 pts/0    00:00:00 grep --color=auto 996
[root@ip-192-169-142-127 ~(keystone_admin)]# netstat -lntp | grep 5000
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      996/python          
[root@ip-192-169-142-127 ~(keystone_admin)]# ps -ef | grep 996
keystone   996     1  0 09:12 ?        00:00:05 /usr/bin/python /usr/bin/keystone-all
keystone  3975   996  0 09:13 ?        00:00:02 /usr/bin/python /usr/bin/keystone-all
keystone  3976   996  0 09:13 ?        00:00:02 /usr/bin/python /usr/bin/keystone-all
keystone  3977   996  0 09:13 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all
keystone  3978   996  0 09:13 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all
root      9513  5073  0 09:30 pts/0    00:00:00 grep --color=auto 996

edit flag offensive delete link

add a comment

answered 2015-05-17 07:37:15 -0500

xinwenzeng

11 ●1 ●2 ●3

I have the same problem about you . I find the port 5000 and 35357 is used by apache netstat -tunlp | grep 35357 tcp6 0 0 :::35357 :::* LISTEN 6784/httpd

apache use port 35357 because: the docs tell us edit /etc/httpd/conf.d/wsgi-keystone.conf Listen 5000 Listen 35357

<virtualhost *:5000=""> WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone display-name=%{GROUP} WSGIProcessGroup keystone-public WSGIScriptAlias / /var/www/cgi-bin/keystone/main WSGIApplicationGroup %{GLOBAL} WSGIPassAuthorization On LogLevel info ErrorLogFormat "%{cu}t %M" ErrorLog /var/log/httpd/keystone-error.log CustomLog /var/log/httpd/keystone-access.log combined </virtualhost>

<virtualhost *:35357=""> WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone display-name=%{GROUP} WSGIProcessGroup keystone-admin WSGIScriptAlias / /var/www/cgi-bin/keystone/admin WSGIApplicationGroup %{GLOBAL} WSGIPassAuthorization On LogLevel info ErrorLogFormat "%{cu}t %M" ErrorLog /var/log/httpd/keystone-error.log CustomLog /var/log/httpd/keystone-access.log combined </virtualhost>

so the openstack-keystone could not start. I do not know how to solve this problem. I just stop apache service then start openstack-keystone,it is ok. But apache is top.

Best regards!

edit flag offensive delete link

add a comment

answered 2015-05-19 10:09:38 -0500

dbaxps

10778 ●57 ●112 ●224 http://bderzhavets.blo...

updated 2015-05-20 09:19:20 -0500

I've just completed RDO Kilo AIO install on real box and see :-

== Keystone service ==
openstack-keystone:                     inactive  (disabled on boot)
================== 

[root@CentOS71WKS ~(keystone_admin)]# netstat -lntp | grep  35357
tcp6       0      0 :::35357                :::*                    LISTEN      14373/httpd         
[root@CentOS71WKS ~(keystone_admin)]# netstat -lntp | grep  500
tcp6       0      0 :::5000                 :::*                    LISTEN      14373/httpd 

[root@CentOS71WKS ~(keystone_admin)]# ps -ef | grep keystone
root      7531 18227  0 18:10 pts/0    00:00:00 grep --color=auto keystone
keystone 14375 14373  0 17:53 ?        00:00:07 keystone-admin  -DFOREGROUND
keystone 14376 14373  0 17:53 ?        00:00:01 keystone-main   -DFOREGROUND

Generated answer-file provide an options ( both of them would work on RDO Kilo)

# Name of service to use to run the Identity service (keystone or
# httpd). ['keystone', 'httpd']
CONFIG_KEYSTONE_SERVICE_NAME=httpd

Any option may be selected for RDO Kilo Multi node deployment, openstack-keystone.service is not required to be started. On RDO Kilo Controller Node :-

[root@ip-192-169-142-127 ~(keystone_admin)]# netstat -lntp |  grep 35357
tcp6       0      0 :::35357                :::*                    LISTEN      3115/httpd          
[root@ip-192-169-142-127 ~(keystone_admin)]# netstat -lntp |  grep 5000
tcp6       0      0 :::5000                 :::*                    LISTEN      3115/httpd          
[root@ip-192-169-142-127 ~(keystone_admin)]# ps -ef | grep 3115
root      3115     1  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
keystone  3126  3115  0 15:19 ?        00:00:43 keystone-admin  -DFOREGROUND
keystone  3128  3115  0 15:19 ?        00:00:04 keystone-main   -DFOREGROUND
apache    3129  3115  0 15:19 ?        00:00:09 /usr/sbin/httpd -DFOREGROUND
apache    3130  3115  0 15:19 ?        00:00:16 /usr/sbin/httpd -DFOREGROUND
apache    3131  3115  0 15:19 ?        00:00:08 /usr/sbin/httpd -DFOREGROUND
apache    3132  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3133  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3136  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3137  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3138  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3139  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3140  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3141  3115  0 15:19 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    3244  3115  0 16:48 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache   24514  3115  0 15:54 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND

edit flag offensive delete link

add a comment

answered 2015-05-20 22:24:41 -0500

Nik
21 ●1 ●2 ●3

I encountered a similar issue with a recent RDO Packstack Juno deployment. After the deployment, Keystone wouldn't start because port 35357 was being used by HTTP.

It turned out Puppet had reserved those ports in /etc/httpd/conf/ports.conf for httpd which was causing keystone to fail on start up because it couldn't bind to port 35357.

By commenting out ports 5000 and 35357 in /etc/httpd/conf/ports.conf and restarting httpd and keystone services, it all seemed to work again. So far the deployment doesn't seem to be having any authentication or dashboard issues with those ports commented out of the ports.conf file.

Hopefully that helps ...

edit flag offensive delete link

Comments

I didn't use packstack installation process.. So shall I change th below file for port 5000 & 35357

/etc/httpd/conf.d/wsgi-keystone.conf

Pradipta_OS_M ( 2015-05-21 21:27:37 -0500 )edit

Can you run the commands below to see what the ports are bound too?

lsof -i -n -P | grep keystone; lsof -i -n -P | grep httpd

Nik ( 2015-05-21 22:13:54 -0500 )edit

add a comment

Keystone socket error "[Errno 98] Address already in use"

5 answers

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Keystone socket error "[Errno 98] Address already in use" edit

5 answers

Comments

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Keystone socket error "[Errno 98] Address already in use"