Periodic error in /var/log/nova/nova-api.log

asked 2017-05-01 12:48:37 -0500

tiger gravatar image

updated 2017-05-01 18:30:59 -0500

About every five seconds, I get this in /var/log/nova/nova-api.log:

 2017-05-01 17:44:42.238 7823 ERROR nova.wsgi [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] Could not bind to 0.0.0.0:8774
2017-05-01 17:44:42.239 7823 CRITICAL nova [req-30be54a7-e047-49f8-94df-be6257216965 - - - - -] error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova Traceback (most recent call last):
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/bin/nova-api", line 10, in <module>
2017-05-01 17:44:42.239 7823 ERROR nova     sys.exit(main())
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/cmd/api.py", line 59, in main
2017-05-01 17:44:42.239 7823 ERROR nova     server = service.WSGIService(api, use_ssl=should_use_ssl)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/service.py", line 334, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     max_url_len=max_url_len)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/nova/wsgi.py", line 101, in __init__
2017-05-01 17:44:42.239 7823 ERROR nova     self._socket = eventlet.listen(bind_addr, family, backlog=backlog)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib/python2.7/site-packages/eventlet/convenience.py", line 43, in listen
2017-05-01 17:44:42.239 7823 ERROR nova     sock.bind(addr)
2017-05-01 17:44:42.239 7823 ERROR nova   File "/usr/lib64/python2.7/socket.py", line 224, in meth
2017-05-01 17:44:42.239 7823 ERROR nova     return getattr(self._sock,name)(*args)
2017-05-01 17:44:42.239 7823 ERROR nova error: [Errno 98] Address already in use
2017-05-01 17:44:42.239 7823 ERROR nova

As an aside, do the developers of Nova expect this above to be useful to anyone? \Anyway, let's look at who or what already has port 8774 open:

[root@kvm8 nova(keystone_admin)]# netstat -pln | grep 8774
tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      8112/python2

Ummmm, OK. So python is running a script and something else is trying to open the same port.

Everything about Openstack just has to be as difficult as possible. Anyone have an idea where to look next? Thanks..

More information

[root@kvm8 nova(keystone_admin)]# service openstack-nova-api restart
Redirecting to /bin/systemctl restart  openstack-nova-api.service
Job for openstack-nova-api.service failed because the control process exited with error code. See "systemctl status openstack-nova-api.service" and "journalctl -xe" for details.


[root@kvm8 nova(keystone_admin)]# systemctl status openstack-nova-api.service
● openstack-nova-api.service - OpenStack Nova API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)
   Active: activating (start) since Mon 2017-05-01 23:29:06 GMT; 2s ago
 Main PID: 1841 (nova-api)
   CGroup: /system.slice/openstack-nova-api.service
           └─1841 /usr/bin/python2 /usr/bin/nova-api

May 01 23:29:06 kvm8.mia.ccncore.net systemd[1]: Starting OpenStack Nova API Server...


[root@kvm8 nova(keystone_admin)]# journalctl -xe
May 01 23:29:35 kvm8.mia.ccncore.net systemd[1]: Unit openstack-nova-api.service entered ...
(more)
edit retag flag offensive close merge delete

Comments

Personally, I think it's very useful to know that some other process uses the same port as Nova. Check what is process 8112. Perhaps you are running several Nova API services.

Bernd Bausch gravatar imageBernd Bausch ( 2017-05-01 17:15:47 -0500 )edit

It logs would be useful if they spoke of: Who it is, what it is doing, the outcome, and what condition failed to bring about success (if failure). Something like "I am Nova. I am trying to fry an egg. I was not able to. The frying pan is missing". Thinks like "reason=reason" are just silly

tiger gravatar imagetiger ( 2017-05-04 08:49:43 -0500 )edit

Nova can't figure this out who blocks the port without having root privileges. "What it's doing" is a bit vague; what would you expect to see? The outcome is that nova-api can't bind to that port. You, as an operator or system administrator, need to troubleshoot this, not Nova.

Bernd Bausch gravatar imageBernd Bausch ( 2017-05-04 10:04:37 -0500 )edit