Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

live migration problem

Hi guys. I am trying to live migrate an instance from nod1 to nod2 and it is failing with the following message.

nod1:

2014-03-27 16:45:21.027 3478 INFO nova.compute.manager [-] Lifecycle event 2 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391 2014-03-27 16:45:21.205 3478 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During sync_power_state the instance has a pending task. Skip. 2014-03-27 16:45:21.341 3478 INFO nova.compute.manager [-] Lifecycle event 3 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391 2014-03-27 16:45:21.350 3478 ERROR nova.virt.libvirt.driver [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Live Migration failure: Domain not found: no domain with matching name 'instance-00000003'

nod2

{"vendor": "AMD", "model": "qemu64", "arch": "x86_64", "features": ["sse4a", "abm", "lahf_lm", "hypervisor", "popcnt"], "topology": {"cores": 1, "threads": 1, "sockets": 1}} 2014-03-27 16:44:15.393 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Called setup_basic_filtering in nwfilter 2014-03-27 16:44:15.394 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Ensuring static filters 2014-03-27 16:44:17.989 414 INFO nova.compute.manager [-] Lifecycle event 0 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391 2014-03-27 16:44:18.601 414 INFO nova.compute.manager [req-0e8d567f-7a2c-49eb-8240-190bdc6abeab None None] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest 2014-03-27 16:44:18.896 414 INFO nova.compute.manager [-] Lifecycle event 1 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391 2014-03-27 16:44:19.220 414 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest

There are no errors in libvirt log on both sides, or any other errors whatsoever. I also get the similar behavior when I try block migration. Here is my setup:

Distro: openSuSE 13.1 
Kernel: 3.11.10-7-default
OpenStack: 2013.2
Libvirt version: 1.2.2-379.2
Qemu version: 1.7.90-221.1
Network: OVS with GRE tunneling
Environment: nested KVM

/etc/libvirt/libvirt.conf:

listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

/etc/sysconfig/libvirtd:

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"
LIBVIRTD_NOFILES_LIMIT=2048

migration related nova.conf stuff:

live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
vncserver_listen = 0.0.0.0

Any help is greatly appreciated.

live migration problem

Hi guys. I am trying to live migrate an instance from nod1 to nod2 and it is failing with the following message.

nod1:

2014-03-27 16:45:21.027 3478 INFO nova.compute.manager [-] Lifecycle event 2 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.205 3478 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During sync_power_state the instance has a pending task. Skip.
2014-03-27 16:45:21.341 3478 INFO nova.compute.manager [-] Lifecycle event 3 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.350 3478 ERROR nova.virt.libvirt.driver [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Live Migration failure: Domain not found: no domain with matching name 'instance-00000003' 

'instance-00000003'

nod2

{"vendor": "AMD", "model": "qemu64", "arch": "x86_64", "features": ["sse4a", "abm", "lahf_lm", "hypervisor", "popcnt"], "topology": {"cores": 1, "threads": 1, "sockets": 1}}
2014-03-27 16:44:15.393 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Called setup_basic_filtering in nwfilter
2014-03-27 16:44:15.394 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Ensuring static filters
2014-03-27 16:44:17.989 414 INFO nova.compute.manager [-] Lifecycle event 0 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:18.601 414 INFO nova.compute.manager [req-0e8d567f-7a2c-49eb-8240-190bdc6abeab None None] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest
2014-03-27 16:44:18.896 414 INFO nova.compute.manager [-] Lifecycle event 1 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:19.220 414 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest

There are no errors in libvirt log on both sides, or any other errors whatsoever. I also get the similar behavior when I try block migration. Here is my setup:

Distro: openSuSE 13.1 
Kernel: 3.11.10-7-default
OpenStack: 2013.2
Libvirt version: 1.2.2-379.2
Qemu version: 1.7.90-221.1
Network: OVS with GRE tunneling
Environment: nested KVM

/etc/libvirt/libvirt.conf:

listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

/etc/sysconfig/libvirtd:

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"
LIBVIRTD_NOFILES_LIMIT=2048

migration related nova.conf stuff:

live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
vncserver_listen = 0.0.0.0

Any help is greatly appreciated.

Why is live migration problemfailing with 'Domain not found'?

Hi guys. I am trying to live migrate an instance from nod1 to nod2 and it is failing with the following message.

nod1:

2014-03-27 16:45:21.027 3478 INFO nova.compute.manager [-] Lifecycle event 2 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.205 3478 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During sync_power_state the instance has a pending task. Skip.
2014-03-27 16:45:21.341 3478 INFO nova.compute.manager [-] Lifecycle event 3 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.350 3478 ERROR nova.virt.libvirt.driver [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Live Migration failure: Domain not found: no domain with matching name 'instance-00000003'

nod2

{"vendor": "AMD", "model": "qemu64", "arch": "x86_64", "features": ["sse4a", "abm", "lahf_lm", "hypervisor", "popcnt"], "topology": {"cores": 1, "threads": 1, "sockets": 1}}
2014-03-27 16:44:15.393 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Called setup_basic_filtering in nwfilter
2014-03-27 16:44:15.394 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Ensuring static filters
2014-03-27 16:44:17.989 414 INFO nova.compute.manager [-] Lifecycle event 0 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:18.601 414 INFO nova.compute.manager [req-0e8d567f-7a2c-49eb-8240-190bdc6abeab None None] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest
2014-03-27 16:44:18.896 414 INFO nova.compute.manager [-] Lifecycle event 1 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:19.220 414 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest

There are no errors in libvirt log on both sides, or any other errors whatsoever. I also get the similar behavior when I try block migration. Here is my setup:

Distro: openSuSE 13.1 
Kernel: 3.11.10-7-default
OpenStack: 2013.2
Libvirt version: 1.2.2-379.2
Qemu version: 1.7.90-221.1
Network: OVS with GRE tunneling
Environment: nested KVM

/etc/libvirt/libvirt.conf:

listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

/etc/sysconfig/libvirtd:

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"
LIBVIRTD_NOFILES_LIMIT=2048

migration related nova.conf stuff:

live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
vncserver_listen = 0.0.0.0

Any help is greatly appreciated.

Why is live migration failing with 'Domain not found'?

Hi guys. I am trying to live migrate an instance from nod1 (stacktest) to nod2 (ctest) and it is failing with the following message.

nod1:

2014-03-27 16:45:21.027 3478 INFO nova.compute.manager [-] Lifecycle event 2 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.205 3478 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During sync_power_state the instance has a pending task. Skip.
2014-03-27 16:45:21.341 3478 INFO nova.compute.manager [-] Lifecycle event 3 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.350 3478 ERROR nova.virt.libvirt.driver [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Live Migration failure: Domain not found: no domain with matching name 'instance-00000003'

nod2

{"vendor": "AMD", "model": "qemu64", "arch": "x86_64", "features": ["sse4a", "abm", "lahf_lm", "hypervisor", "popcnt"], "topology": {"cores": 1, "threads": 1, "sockets": 1}}
2014-03-27 16:44:15.393 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Called setup_basic_filtering in nwfilter
2014-03-27 16:44:15.394 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Ensuring static filters
2014-03-27 16:44:17.989 414 INFO nova.compute.manager [-] Lifecycle event 0 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:18.601 414 INFO nova.compute.manager [req-0e8d567f-7a2c-49eb-8240-190bdc6abeab None None] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest
2014-03-27 16:44:18.896 414 INFO nova.compute.manager [-] Lifecycle event 1 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:19.220 414 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest

There are no errors in libvirt log on both sides, or any other errors whatsoever. I also get the similar behavior when I try block migration. Here is my setup:

Distro: openSuSE 13.1 
Kernel: 3.11.10-7-default
OpenStack: 2013.2
Libvirt version: 1.2.2-379.2
Qemu version: 1.7.90-221.1
Network: OVS with GRE tunneling
Environment: nested KVM

/etc/libvirt/libvirt.conf:

listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

/etc/sysconfig/libvirtd:

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"
LIBVIRTD_NOFILES_LIMIT=2048

migration related nova.conf stuff:

live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
vncserver_listen = 0.0.0.0

Any help is greatly appreciated.

Why is live migration failing with 'Domain not found'?

Hi guys. I am trying to live migrate an instance from nod1 (stacktest) to nod2 (ctest) and it is failing with the following message.

nod1:

2014-03-27 16:45:21.027 3478 INFO nova.compute.manager [-] Lifecycle event 2 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.205 3478 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During sync_power_state the instance has a pending task. Skip.
2014-03-27 16:45:21.341 3478 INFO nova.compute.manager [-] Lifecycle event 3 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:45:21.350 3478 ERROR nova.virt.libvirt.driver [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Live Migration failure: Domain not found: no domain with matching name 'instance-00000003'

nod2

{"vendor": "AMD", "model": "qemu64", "arch": "x86_64", "features": ["sse4a", "abm", "lahf_lm", "hypervisor", "popcnt"], "topology": {"cores": 1, "threads": 1, "sockets": 1}}
2014-03-27 16:44:15.393 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Called setup_basic_filtering in nwfilter
2014-03-27 16:44:15.394 414 INFO nova.virt.libvirt.firewall [req-c39dd3e3-8502-4ed8-9dc6-cb79289635d8 748d046631224150acbb31deda725d50 d42f7f63f51340af9a80f6d5554ef20c] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] Ensuring static filters
2014-03-27 16:44:17.989 414 INFO nova.compute.manager [-] Lifecycle event 0 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:18.601 414 INFO nova.compute.manager [req-0e8d567f-7a2c-49eb-8240-190bdc6abeab None None] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest
2014-03-27 16:44:18.896 414 INFO nova.compute.manager [-] Lifecycle event 1 on VM e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391
2014-03-27 16:44:19.220 414 INFO nova.compute.manager [-] [instance: e8ae2a2c-3bdf-4322-ba2b-feb0d13ba391] During the sync_power process the instance has moved from host ctest to host stacktest

There are no errors in libvirt log on both sides, or any other errors whatsoever. I also get the similar behavior when I try block migration. Here is my setup:

Distro: openSuSE 13.1 
Kernel: 3.11.10-7-default
OpenStack: 2013.2
Libvirt version: 1.2.2-379.2
Qemu version: 1.7.90-221.1
Network: OVS with GRE tunneling
Environment: nested KVM

/etc/libvirt/libvirt.conf:

listen_tls = 0
listen_tcp = 1
auth_tcp = "none"

/etc/sysconfig/libvirtd:

LIBVIRTD_CONFIG=/etc/libvirt/libvirtd.conf
LIBVIRTD_ARGS="--listen"
LIBVIRTD_NOFILES_LIMIT=2048

migration related nova.conf stuff:

live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
vncserver_listen = 0.0.0.0

Any help is greatly appreciated.

EDIT I am attaching openstack and libvirt debug logs.

Nova logs:

Source nod (stacktest): http://pastebin.com/vidp3QUA
Destination nod (compute): http://pastebin.com/XF5BJvmh

Libvirt logs:

Source nod (stacktest): http://pastebin.com/GgB0iZET
Destination nod (ctest): http://pastebin.com/qhNYJY7B

It seems that live migration is failing because of the following libvirt error:

"error": {"class": "DeviceNotActive", "desc": "No active block job on device 'drive-virtio-disk0'"}}

I am unaware what is causing it and I am still unable to find a solution.