live migration failing with "Unexpected API Error"

asked 2017-08-06 17:07:19 -0500

pacman gravatar image

updated 2017-08-08 09:10:10 -0500

Hi

I installed openstack according to the instructions in the official getting started guide. The Stack is working mostly except for some hickups. One of them are failing live migrations.

Everytime I try to live migrate an instance it fails with "Unexpected API Error". Here is my /var/log/nova/nova-api.log:

2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions [req-585db0ac-b3ef-4e07-afb5-914ad9005ae5 
735e4dd64a0e451bbb5254950af6e6ad f9873284dd3b40edbf9564703c0903ad - default default] Unexpected exception in API method
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 100, in _migrate_live
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     disk_over_commit, host, force, async)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 170, in inner
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 178, in _wrapped
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return fn(self, context, instance, *args, **kwargs)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 151, in inner
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3903, in live_migrate
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     messaging_timeout)
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-06 23:27:21.011 9550 ERROR nova.api.openstack.extensions     self.force_reraise()
2017-08-06 23 ...
(more)
edit retag flag offensive close merge delete

Comments

Hi, how many compute nodes you have? what are the configuration changes you did to setup vm live migration ? BTW check the status of firewall/ iptables. share the output of cat /etc/hosts and getenforce commands

Praveen N gravatar imagePraveen N ( 2017-08-08 04:34:16 -0500 )edit

Hi, i got 5 absolutely identical compute nodes. honestly at first i did not do any modifications - i just expected it to work when following the official setup.

pacman gravatar imagepacman ( 2017-08-08 09:06:15 -0500 )edit

meanwhile i found some older manuals (2013) stating that some libvirt settings have to be adjusted: https://kimizhang.wordpress.com/2013/08/26/openstack-vm-live-migration/ (https://kimizhang.wordpress.com/2013/...) however i did not try it yet since i found out that neither cold migration nor resize are working as well (working on that now)

pacman gravatar imagepacman ( 2017-08-08 09:07:30 -0500 )edit

i am just about to reinstall the cluster but i can say for sure that selinux is off - i added the hosts file in my original post as the comment size is to limited

pacman gravatar imagepacman ( 2017-08-08 09:10:40 -0500 )edit

Have you find a way to solve this problem? I'm facing the exact same issue!

angelvargas gravatar imageangelvargas ( 2019-07-01 08:11:32 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2020-08-06 11:27:29 -0500

Andrew Bogott gravatar image

I was seeing this problem recently; the issue turned out to be a timeout in the HAproxy in front of the nova api. Nova takes quite a while to muster a response to a migration request; when I changed my server timeout from 50s to 120s I started getting useful responses from my migration calls.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2017-08-06 17:07:19 -0500

Seen: 158 times

Last updated: Aug 08 '17