Ask Your Question
0

Live Migration failure: internal error: unexpected migration status in cancelling

asked 2017-06-27 04:34:30 -0500

VincentSwiss gravatar image

Hello everyone,

I'm getting the following error when live migrating instances from an host to another:

Live Migration failure: internal error: unexpected migration status in cancelling

To start the live migration I use the following command:

openstack server migrate --block-migration --disk-overcommit 509aa926-f78e-44fc-b70d-01daaaadddff5 --live compute22.nest.org --wait

The migration fails randomly, sometimes at 20%, others at 80%... If I execute the migration command multiple times the VM will eventually be migrated.

I've tried to increase the nova log level but the error doesn't get any clearer.

Any ideas on how to get more information on this error or how to solve it?

Thank you and best regards,

Vincenzo

edit retag flag offensive close merge delete

Comments

Check the nova-compute log files on the two compute nodes, and the conductor log on the controller. Also libvirt logs if you have any. If not, enable them.

Bernd Bausch gravatar imageBernd Bausch ( 2017-06-27 04:40:29 -0500 )edit

As far as I can see this is the only error message I have in the libvirtd logs: https://paste.ubuntu.com/24963131/ Nova logs just say that the migration was stuck and so it got aborted: https://paste.ubuntu.com/24963138/

VincentSwiss gravatar imageVincentSwiss ( 2017-06-27 07:45:52 -0500 )edit
add a comment

1 answer

Sort by ยป oldest newest most voted
0

answered 2017-06-27 19:05:15 -0500

Bernd Bausch gravatar image

updated 2017-06-28 01:26:01 -0500

Nova aborts your migration because it is taking too long. Most probably your instance is too busy: Nova has transferred a memory page, the instance modifies that page so that it has to be copied again, and so on until Nova gives up.

Other factors are the sheer size of the instance and the capacity of the network used for migration. Which is, by default, the management network.

Recommendations:

  • Calm your instance down
  • Improve networking
  • Use config parameters to influence Nova's migration behaviour; see the admin guide for instructions. A caveat: It depends on your OpenStack release to what extent these instructions are applicable. Live Migration has improved a lot in the last two years.
edit flag offensive delete link more

Comments

Hey there, thank you for you reply. FYI I'm using Mitaka, but it looks like it is as you say, nova kills the migration due to some timeout. I will try to improve this by modifying some config parameter as suggested in the page you linked. Best regards

VincentSwiss gravatar imageVincentSwiss ( 2017-06-28 01:49:16 -0500 )edit

Mitaka implements the timeouts, but post-copy and auto-convergence were added in Newton.

Bernd Bausch gravatar imageBernd Bausch ( 2017-06-28 05:55:26 -0500 )edit
add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2017-06-27 04:34:30 -0500

Seen: 553 times

Last updated: Jun 28 '17

Related questions

Disable IP forwarding on compute nodes [closed]

Nova VM instances files change ownership to root after live migration

Live-migration only works for instances launched by 'Boot from image (create new volume)'

how to use esx as hypervisor

caught error: Timeout while waiting on RPC response - topic: "network", RPC method: "validate_networks" info: "<unknown>"

Verify communication in chain network topology

heartbeats for compute nodes?

nova-api really slow to respond

Standalone mode (no network)

how to allocate ext IP to an instance