Unable to perform livemigration

asked 2020-08-10 14:51:26 -0500

EddyNava gravatar image

updated 2020-08-12 00:47:51 -0500

eblock gravatar image

2020-08-10 13:38:42.688 3661 ERROR nova.virt.libvirt.driver [-] [instance: d0cae88e-7039-4f73-9077-368fbdde0a50] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+tcp://openstack-comp02/system: unable to connect to server at ‘openstack-comp02:16509’: Connection refused: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://openstack-comp02/system: unable to connect to server at ‘openstack-comp02:16509’: Connection refused

The port 16509 is down on both nodes and I'm not able to set them up

This is happening on Openstack Train, I'm not able to perform live migration from node 1 to node 2, I used to work with Newton and I was able to set it up on libvirt-bin but I'm not able to find it on Train

Compute 1 output of systemctl status libvirtd

root@openstack-comp01:~# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-08-03 12:23:35 CDT; 1 weeks 1 days ago
     Docs: man:libvirtd(8)
 Main PID: 2517 (libvirtd)
    Tasks: 282 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─  2517 /usr/sbin/libvirtd
           ├─  3260 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─  3261 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─  4218 /usr/bin/qemu-system-x86_64 -name guest=instance-0000003f,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-0000003f/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
           ├─  5419 /usr/bin/qemu-system-x86_64 -name guest=instance-0000002b,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-instance-0000002b/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
           ├─ 11379 /usr/bin/qemu-system-x86_64 -name guest=instance-00000049,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-15-instance-00000049/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
           ├─ 11842 /usr/bin/qemu-system-x86_64 -name guest=instance-0000004c,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-16-instance-0000004c/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
           ├─ 14719 /usr/bin/qemu-system-x86_64 -name guest=instance-0000004d,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-19-instance-0000004d/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
           ├─ 23700 /usr/bin/qemu-system-x86_64 -name guest=instance-00000060,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-instance-00000060/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
           ├─101753 /usr/bin/qemu-system-x86_64 -name guest=instance-0000004e,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-instance-0000004e/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
           └─758342 /usr/bin/qemu-system-x86_64 -name guest=instance-000000e2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-50-instance-000000e2/master-key.aes -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=off -cp
Aug ...
edit retag flag offensive close merge delete


Please show the output of systemctl status libvirtd. Also search syslog for any hints why libvirt fails to start.

eblock gravatar imageeblock ( 2020-08-11 02:10:21 -0500 )edit

Hi @eblock I just updated the question with the output you asked for.

EddyNava gravatar imageEddyNava ( 2020-08-11 17:18:31 -0500 )edit

Have you tried this?

eblock gravatar imageeblock ( 2020-08-12 00:50:00 -0500 )edit

I did, I use to have newton version, and libvirt-bin existed back then, and would just modify libvirtd_opts=" -d" to libvirtd_opts=" -d -l" then would run stop libvirt-bin && start libvirt-bin then ps -ef | grep libvirt

But this settings are not showing on Train version.

EddyNava gravatar imageEddyNava ( 2020-08-12 16:24:24 -0500 )edit

I'm not familiar with libvirt-bin. Can you try to set this in /etc/libvirt/libvirtd.conf as suggested in the link I mentioned? Set listen_tcp = 1 and tcp_port = "16509" and restart libvirtd. Check with lsof -i -P | grep 16509 if the port is open.

eblock gravatar imageeblock ( 2020-08-13 04:35:38 -0500 )edit