Unexplained intermittent failure in horizon dashboard

asked 2019-08-16 09:48:44 -0500

josh.wilson gravatar image

Hello OpenStack community!

I'm running into a strange problem with my horizon dashboard. I'm running OpenStack Stein, on an Ubuntu Desktop 18.04 LTS controller. I was troubleshooting a cinder volume that wouldn't delete, which took my into the MariaDB database. Now, whenever I browse to the project/instances page it gives me a failure 9 out of 10 times. When I tail the apache2 error.log this is the output that I get. When it does successfully load, there is no output in the apache error log.tail -f

I've been pouring through my cinder database trying to find anything that lines up with boot_volume, volume-image_metadata, or image_id. I can't find anything that lines up. I was following the instructions from this answer when it happened, but going through the cinder.volumes table hasn't netted me anything.

https://ask.openstack.org/en/question/66918/how-to-delete-volume-with-available-status-and-attached-to/

Thank you for the help!

[Fri Aug 16 09:51:19.418540 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527] ERROR django.request Internal Server Error: /horizon/project/instances/
[Fri Aug 16 09:51:19.418595 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527] Traceback (most recent call last):
[Fri Aug 16 09:51:19.418599 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 41, in inner
[Fri Aug 16 09:51:19.418603 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = get_response(request)
[Fri Aug 16 09:51:19.418606 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
[Fri Aug 16 09:51:19.418608 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = self.process_exception_by_middleware(e, request)
[Fri Aug 16 09:51:19.418611 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 185, in _get_response
[Fri Aug 16 09:51:19.418614 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Aug 16 09:51:19.418617 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/horizon/decorators.py", line 36, in dec
[Fri Aug 16 09:51:19.418620 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     return view_func(request, *args, **kwargs)
[Fri Aug 16 09:51:19.418623 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/horizon/decorators.py", line 52, in dec
[Fri Aug 16 09:51:19.418625 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     return view_func(request, *args, **kwargs ...
(more)
edit retag flag offensive close merge delete

Comments

The issue is with the id 3369ffe8-dbf6-4a3f-ad2f-7eb35d7355f4. You may need to clean this up from the database to get rid of the error.

Syed Awais Ali gravatar imageSyed Awais Ali ( 2019-08-21 03:53:20 -0500 )edit

Thank you for the answer!

My end goal was to track down that ID somewhere in the database by looking for the relevant tables. Based on what this error says, do you have any insight as to where it may be?

josh.wilson gravatar imagejosh.wilson ( 2019-08-21 07:50:09 -0500 )edit

You may need to eliminate all the references to the said ID. You need to play with the database. just like this link refers to the image model and has id. So you may need to look into glance for Images id.

Syed Awais Ali gravatar imageSyed Awais Ali ( 2019-08-22 03:47:02 -0500 )edit

While this hasn't fully resolved the issue yet. It led me down a path to be able to find where this key existed. To maybe help someone else who comes across it, I used this command extensively.

mysqldump cinder --extended-insert=FALSE | grep "3369ffe8-dbf6-4a3f-ad2f-7eb35d7355f4" | less -S
josh.wilson gravatar imagejosh.wilson ( 2019-08-22 15:32:53 -0500 )edit