Migration between compute nodes fails

asked 2014-09-10 11:18:56 -0600

JS gravatar image

updated 2014-09-11 03:02:39 -0600

I'm running icehouse on Ubuntu 14.04 LTS, installed from the Ubuntu repositories.

I have the following nodes:

  • stack-man (controller)
  • stack-net (network)
  • stack-srv1,stack-srv2 (2 x compute)

Virtualisation plugin is KVM/Libvirt. Networking is via neutron using ml2 plugin with gre tunnels & openvswitch.

I have 3 networks - all nodes have an interface on the management network (their primary names, as above). The network and compute nodes each have an interface on the vm tunnel network (as configured and working in neutron, their hostnames are suffixed with -vm; stack-net-vm, stack-srv1-vm, etc.). Finally there is a storage traffic network for stack-man, stack-srv1/2 and the GlusterFS nodes (hostnames suffixed with -fs; stack-man-fs, stack-srv1-fs etc).

Migration and management commands are therefore only running over the management network.

Instance storage is provisioned by a GlusterFS mount on the controller and compute nodes (/var/lib/nova/instances is a link to the 'instance' brick from the GlusterFS nodes) provisioned by a pair of storage nodes. The storage nodes don't run any Openstack services.

Block storage is currently running on the controller node also - using the GlusterfsDriver in /etc/cinder/cinder.conf with the cinder volume configured in /etc/cinder/glusterfs.conf (using the 'cinder' brick from the GlusterFS nodes). At some point I'll move additional cinder-volume services to the compute nodes for resiliency.

Everything has been setup pretty much according to the install guide - I had some issues configuring libvirt to do remote access, but that all seems to work (certs all installed - virsh works from either the controller or compute nodes).

I've created an instance from the CirrOs image - it provisions a new volume, mounts it, creates an instance folder, starts the vm, I can login, I get DHCP assigned networking from neutron etc. All great. However, neither migrate nor live-migration features work.

/var/log/nova/nova-compute.log lists the following message regardless of whether migrate or live-migration is used:

'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\\n\'

All the usual command line interface tools work - nova list/stop/start/show, neutron agent-list, cinder list etc.

An actual trace of the sequence of actions for a non-live migration, debug=True enabled in nova.conf:

root@stack-man:/var/lib/nova/instances# nova start 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances# nova show 84b14df7-7e29-49cc-bc9a-5f92ab050639
| Property                             | Value                                                    |
| DemoNet1 network                     |                                              |
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | stack-srv1                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | stack-srv1                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000012                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-09-10T15:36:02.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2014-09-10T15:35:53Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 373a2ec65e77c9d55e76e6006cf40bf142fcffa471ff3dcfcff21d03 |
| id                                   | 84b14df7-7e29-49cc-bc9a-5f92ab050639                     |
| image                                | Attempt to boot from volume - no image supplied          |
| key_name                             | demo-key                                                 |
| metadata                             | {}                                                       |
| name                                 | Cirros1                                                  |
| os-extended-volumes:volumes_attached | [{"id": "5726609f-4e58-47bc-baf6-995a7cafdc1f"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | cb9ab00c01c44a85989d1886374e20e5                         |
| updated                              | 2014-09-10T15:53:37Z                                     |
| user_id                              | 4edeb1fc16dd415eb9225f492d2d4f59                         |
root@stack-man:/var/lib/nova/instances# nova migrate 84b14df7-7e29-49cc-bc9a-5f92ab050639

Then looking at /var/log/nova-compute.log on stack-srv2 shows ... (more)

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted

answered 2014-09-10 17:03:16 -0600

rahulrajvn gravatar image

Enable true live migration

By default, the Compute service does not use the libvirt live migration functionality. To enable this functionality, add the following line to the nova.conf file:


And also make sure following changes are done in compute nodes.

sed -i "s/#listen_tls.*/listen_tls=0/g" /etc/libvirt/libvirtd.conf

sed -i "s/#tcp_port.*/tcp_port =\"16509\"/g" /etc/libvirt/libvirtd.conf

sed -i "s/#tls_port.*/tls_port =\"16514\"/g" /etc/libvirt/libvirtd.conf

sed -i "s/#listen_tcp.*/listen_tcp=1/g" /etc/libvirt/libvirtd.conf

echo "auth_tcp=\"none\" " >> /etc/libvirt/libvirtd.conf

sed -i "s/#LIBVIRTD_ARGS.*/LIBVIRTD_ARGS=\"--listen\"/g" /etc/sysconfig/libvirtd

Restart the service

/etc/init.d/libvirtd restart

openstack-service restart

edit flag offensive delete link more


Updated my question - those are all already set. Connections from 'virsh' work to each node. But both non-live and live migrations fail, so it can't just be the live migration config that is at fault.

JS gravatar imageJS ( 2014-09-11 03:23:42 -0600 )edit

just to try can u try setting password less authentication between nova user's between controller/compute nodes.

rahulrajvn gravatar imagerahulrajvn ( 2014-09-11 06:23:17 -0600 )edit

Please confirm, I do not have /etc/sysconfig/libvirtd file, nor do I have a /etc/sysconfig directory. Is this created or should be there by default?

bobyakov gravatar imagebobyakov ( 2015-05-28 10:48:46 -0600 )edit

answered 2014-09-25 10:39:13 -0600

I ran into this same issue under the same circumstances. Ubuntu 14.04 LTS, Icehouse, KVM/libvirt, Neutron using OVS and GRE tunneling.

While the authorization error shows up in the nova-compute logs, the actual problem - for me, at least - was an incorrect neutron_admin_password in /etc/nova/nova.conf on the controller node.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-09-10 11:18:56 -0600

Seen: 2,487 times

Last updated: Sep 25 '14