Ask Your Question
0

Live Migration failure: internal error: unexpected migration status in cancelling

asked 2017-06-27 04:34:30 -0600

VincentSwiss gravatar image

Hello everyone,

I'm getting the following error when live migrating instances from an host to another:

Live Migration failure: internal error: unexpected migration status in cancelling

To start the live migration I use the following command:

openstack server migrate --block-migration --disk-overcommit 509aa926-f78e-44fc-b70d-01daaaadddff5 --live compute22.nest.org --wait

The migration fails randomly, sometimes at 20%, others at 80%... If I execute the migration command multiple times the VM will eventually be migrated.

I've tried to increase the nova log level but the error doesn't get any clearer.

Any ideas on how to get more information on this error or how to solve it?

Thank you and best regards,

Vincenzo

edit retag flag offensive close merge delete

Comments

Check the nova-compute log files on the two compute nodes, and the conductor log on the controller. Also libvirt logs if you have any. If not, enable them.

Bernd Bausch gravatar imageBernd Bausch ( 2017-06-27 04:40:29 -0600 )edit

As far as I can see this is the only error message I have in the libvirtd logs: https://paste.ubuntu.com/24963131/ Nova logs just say that the migration was stuck and so it got aborted: https://paste.ubuntu.com/24963138/

VincentSwiss gravatar imageVincentSwiss ( 2017-06-27 07:45:52 -0600 )edit
add a comment

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-06-27 19:05:15 -0600

Bernd Bausch gravatar image

updated 2017-06-28 01:26:01 -0600

Nova aborts your migration because it is taking too long. Most probably your instance is too busy: Nova has transferred a memory page, the instance modifies that page so that it has to be copied again, and so on until Nova gives up.

Other factors are the sheer size of the instance and the capacity of the network used for migration. Which is, by default, the management network.

Recommendations:

  • Calm your instance down
  • Improve networking
  • Use config parameters to influence Nova's migration behaviour; see the admin guide for instructions. A caveat: It depends on your OpenStack release to what extent these instructions are applicable. Live Migration has improved a lot in the last two years.
edit flag offensive delete link more

Comments

Hey there, thank you for you reply. FYI I'm using Mitaka, but it looks like it is as you say, nova kills the migration due to some timeout. I will try to improve this by modifying some config parameter as suggested in the page you linked. Best regards

VincentSwiss gravatar imageVincentSwiss ( 2017-06-28 01:49:16 -0600 )edit

Mitaka implements the timeouts, but post-copy and auto-convergence were added in Newton.

Bernd Bausch gravatar imageBernd Bausch ( 2017-06-28 05:55:26 -0600 )edit
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2017-06-27 04:34:30 -0600

Seen: 701 times

Last updated: Jun 28 '17

Related questions

nova compute

live-migration - libvirt - invtsc

Unable to start openstack-nova-compute.service

Nova-compute does not show up in service list

nova-network create on compute node

Amazon EMR vs OpenStack

Connection to libvirt failed

List instances project wise?

compute instance

ping running instance