Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Periodic error in /var/log/nova/nova-api.log

About every five seconds, I get this in /var/log/nova/nova-api.log:

 2017-05-01 17:44:42.238 7823 ERROR nova.wsgi [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] Could not bind to 0.0.0.0:8774
2017-05-01 17:44:42.239 7823 CRITICAL nova [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova Traceback (most recent call last):
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/bin/nova-api", line 10, in <module>
2017-05-01 17:44:42.239 7823 ERROR nova     sys.exit(main())
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 59, in main
2017-05-01 17:44:42.239 7823 ERROR nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/service.py", line 334, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     max_url_len=max_url_len)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     self._socket = eventlet.listen(bind_addr, family, backlog=backlog)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2017-05-01 17:44:42.239 7823 ERROR nova     sock.bind(addr)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2017-05-01 17:44:42.239 7823 ERROR nova     return getattr(self._sock,name)(*args)
2017-05-01 17:44:42.239 7823 ERROR nova error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova

As an aside, do the developers of Nova expect this above to be useful to anyone? \Anyway, let's look at who or what already has port 8774 open:

[root@kvm8 nova(keystone_admin)]# netstat -pln | grep 8774
tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      8112/python2

Ummmm, OK. So python is running a script and something else is trying to open the same port.

Everything about Openstack just has to be as difficult as possible. Anyone have an idea where to look next? Thanks..

Periodic error in /var/log/nova/nova-api.log

About every five seconds, I get this in /var/log/nova/nova-api.log:

 2017-05-01 17:44:42.238 7823 ERROR nova.wsgi [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] Could not bind to 0.0.0.0:8774
2017-05-01 17:44:42.239 7823 CRITICAL nova [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova Traceback (most recent call last):
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/bin/nova-api", line 10, in <module>
2017-05-01 17:44:42.239 7823 ERROR nova     sys.exit(main())
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 59, in main
2017-05-01 17:44:42.239 7823 ERROR nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/service.py", line 334, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     max_url_len=max_url_len)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     self._socket = eventlet.listen(bind_addr, family, backlog=backlog)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2017-05-01 17:44:42.239 7823 ERROR nova     sock.bind(addr)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2017-05-01 17:44:42.239 7823 ERROR nova     return getattr(self._sock,name)(*args)
2017-05-01 17:44:42.239 7823 ERROR nova error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova

As an aside, do the developers of Nova expect this above to be useful to anyone? \Anyway, let's look at who or what already has port 8774 open:

[root@kvm8 nova(keystone_admin)]# netstat -pln | grep 8774
tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      8112/python2

Ummmm, OK. So python is running a script and something else is trying to open the same port.

Everything about Openstack just has to be as difficult as possible. Anyone have an idea where to look next? Thanks..

More information

[root@kvm8 nova(keystone_admin)]# service openstack-nova-api restart
Redirecting to /bin/systemctl restart  openstack-nova-api.service
Job for openstack-nova-api.service failed because the control process exited with error code. See "systemctl status openstack-nova-api.service" and "journalctl -xe" for details.


[root@kvm8 nova(keystone_admin)]# systemctl status openstack-nova-api.service
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: activating (start) since Mon 2017-05-01 23:29:06 GMT; 2s ago
 Main PID: 1841 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           └─1841 /usr/bin/python2 /usr/bin/nova-api

May 01 23:29:06 kvm8.mia.ccncore.net systemd[1]: Starting OpenStack Nova API Server...


[root@kvm8 nova(keystone_admin)]# journalctl -xe
May 01 23:29:35 kvm8.mia.ccncore.net systemd[1]: Unit openstack-nova-api.service entered failed state.
May 01 23:29:35 kvm8.mia.ccncore.net systemd[1]: openstack-nova-api.service failed.
May 01 23:29:35 kvm8.mia.ccncore.net systemd[1]: openstack-nova-api.service holdoff time over, scheduling restart.
May 01 23:29:35 kvm8.mia.ccncore.net systemd[1]: Starting OpenStack Nova API Server...
-- Subject: Unit openstack-nova-api.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-api.service has begun starting up.
May 01 23:29:37 kvm8.mia.ccncore.net account-server[29395]: 192.168.140.218 - - [01/May/2017:23:29:37 +0000] "HEAD /swiftloopback/136425/.expiring_objects" 404 - "HEAD http://localhost/v1/.expiring
May 01 23:29:37 kvm8.mia.ccncore.net object-expirer[27952]: Pass beginning; 0 possible containers; 0 possible objects (txn: tx6c5cf9a596e841a58b0e5-005907c4e1)
May 01 23:29:37 kvm8.mia.ccncore.net account-server[29393]: 192.168.140.218 - - [01/May/2017:23:29:37 +0000] "GET /swiftloopback/136425/.expiring_objects" 404 - "GET http://localhost/v1/.expiring_o
May 01 23:29:37 kvm8.mia.ccncore.net object-expirer[27952]: Pass completed in 0s; 0 objects expired (txn: tx671560b2fd60433686852-005907c4e1)
May 01 23:29:38 kvm8.mia.ccncore.net object-server[29970]: Starting object replication pass.
May 01 23:29:38 kvm8.mia.ccncore.net object-server[29970]: Nothing replicated for 0.0198581218719 seconds.
May 01 23:29:38 kvm8.mia.ccncore.net object-server[29970]: Object replication complete. (0.00 minutes)
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: openstack-nova-api.service: main process exited, code=exited, status=1/FAILURE
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: Failed to start OpenStack Nova API Server.
-- Subject: Unit openstack-nova-api.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-api.service has failed.
-- 
-- The result is failed.
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: Unit openstack-nova-api.service entered failed state.
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: openstack-nova-api.service failed.
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: openstack-nova-api.service holdoff time over, scheduling restart.
May 01 23:29:40 kvm8.mia.ccncore.net systemd[1]: Starting OpenStack Nova API Server...
-- Subject: Unit openstack-nova-api.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-api.service has begun starting up.