Unexplained intermittent failure in horizon dashboard

asked 2019-08-16 09:48:44 -0500

josh.wilson gravatar image

Hello OpenStack community!

I'm running into a strange problem with my horizon dashboard. I'm running OpenStack Stein, on an Ubuntu Desktop 18.04 LTS controller. I was troubleshooting a cinder volume that wouldn't delete, which took my into the MariaDB database. Now, whenever I browse to the project/instances page it gives me a failure 9 out of 10 times. When I tail the apache2 error.log this is the output that I get. When it does successfully load, there is no output in the apache error log.tail -f

I've been pouring through my cinder database trying to find anything that lines up with boot_volume, volume-image_metadata, or image_id. I can't find anything that lines up. I was following the instructions from this answer when it happened, but going through the cinder.volumes table hasn't netted me anything.

https://ask.openstack.org/en/question/66918/how-to-delete-volume-with-available-status-and-attached-to/

Thank you for the help!

[Fri Aug 16 09:51:19.418540 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527] ERROR django.request Internal Server Error: /horizon/project/instances/
[Fri Aug 16 09:51:19.418595 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527] Traceback (most recent call last):
[Fri Aug 16 09:51:19.418599 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 41, in inner
[Fri Aug 16 09:51:19.418603 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = get_response(request)
[Fri Aug 16 09:51:19.418606 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
[Fri Aug 16 09:51:19.418608 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = self.process_exception_by_middleware(e, request)
[Fri Aug 16 09:51:19.418611 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 185, in _get_response
[Fri Aug 16 09:51:19.418614 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Aug 16 09:51:19.418617 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/horizon/decorators.py", line 36, in dec
[Fri Aug 16 09:51:19.418620 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     return view_func(request, *args, **kwargs)
[Fri Aug 16 09:51:19.418623 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]   File "/usr/lib/python3/dist-packages/horizon/decorators.py", line 52, in dec
[Fri Aug 16 09:51:19.418625 2019] [wsgi:error] [pid 22042:tid 139727399786240] [remote 192.168.10.12:50527]     return view_func(request, *args, **kwargs ...
(more)
edit retag flag offensive close merge delete

Comments

The issue is with the id 3369ffe8-dbf6-4a3f-ad2f-7eb35d7355f4. You may need to clean this up from the database to get rid of the error.

Syed Awais Ali gravatar imageSyed Awais Ali ( 2019-08-21 03:53:20 -0500 )edit

Thank you for the answer!

My end goal was to track down that ID somewhere in the database by looking for the relevant tables. Based on what this error says, do you have any insight as to where it may be?

josh.wilson gravatar imagejosh.wilson ( 2019-08-21 07:50:09 -0500 )edit

You may need to eliminate all the references to the said ID. You need to play with the database. just like this link refers to the image model and has id. So you may need to look into glance for Images id.

Syed Awais Ali gravatar imageSyed Awais Ali ( 2019-08-22 03:47:02 -0500 )edit

While this hasn't fully resolved the issue yet. It led me down a path to be able to find where this key existed. To maybe help someone else who comes across it, I used this command extensively.

mysqldump cinder --extended-insert=FALSE | grep "3369ffe8-dbf6-4a3f-ad2f-7eb35d7355f4" | less -S
josh.wilson gravatar imagejosh.wilson ( 2019-08-22 15:32:53 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2019-10-11 14:17:35 -0500

josh.wilson gravatar image

To the credit of Syed, he had the answer. I've been so delayed in documenting an answer because I've been procrastinating on finishing it. As anyone who has spent anytime in a production DB on OpenStack knows, the web of links is extensive. So, after a few hours of pouring through MariaDB to eliminate references to that ID I had a realization. It would be faster for me to rebuild the three servers that I had left on a different baseline image. So instead of his recommended approach of purging that ID from the database, I instead purged the instances that used it from OpenStack. So technically speaking I still have a bad image reference metadata in my database, but nothing uses it now so my problem has gone away.

edit flag offensive delete link more

Comments

Great !! This is a good workaround to keep things moving.

Syed Awais Ali gravatar imageSyed Awais Ali ( 2019-10-13 07:17:58 -0500 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-08-16 09:48:44 -0500

Seen: 91 times

Last updated: Aug 16 '19