Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Migration between compute nodes fails

I'm running icehouse on Ubuntu 14.04 LTS, installed from the Ubuntu repositories.

I have the following nodes:

  • stack-man (controller)
  • stack-net (network)
  • stack-srv1,stack-srv2 (2 x compute)

Virtualisation plugin is KVM/Libvirt. Networking is via neutron using ml2 plugin with gre tunnels & openvswitch.

I have 3 networks - all nodes have an interface on the management network (their primary names, as above). The network and compute nodes each have an interface on the vm tunnel network (as configured and working in neutron, their hostnames are suffixed with -vm; stack-net-vm, stack-srv1-vm, etc.). Finally there is a storage traffic network for stack-man, stack-srv1/2 and the GlusterFS nodes (hostnames suffixed with -fs; stack-man-fs, stack-srv1-fs etc).

Migration and management commands are therefore only running over the management network.

Instance storage is provisioned by a GlusterFS mount on the controller and compute nodes (/var/lib/nova/instances is a link to the 'instance' brick from the GlusterFS nodes) provisioned by a pair of storage nodes. The storage nodes don't run any Openstack services.

Block storage is currently running on the controller node also - using the GlusterfsDriver in /etc/cinder/cinder.conf with the cinder volume configured in /etc/cinder/glusterfs.conf (using the 'cinder' brick from the GlusterFS nodes). At some point I'll move additional cinder-volume services to the compute nodes for resiliency.

Everything has been setup pretty much according to the install guide - I had some issues configuring libvirt to do remote access, but that all seems to work (certs all installed - virsh works from either the controller or compute nodes).

I've created an instance from the CirrOs image - it provisions a new volume, mounts it, creates an instance folder, starts the vm, I can login, I get DHCP assigned networking from neutron etc. All great. However, neither migrate nor live-migration features work.

/var/log/nova/nova-compute.log lists the following message regardless of whether migrate or live-migration is used:

'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\\n\'

All the usual command line interface tools work - nova list/stop/start/show, neutron agent-list, cinder list etc.

An actual trace of the sequence of actions for a non-live migration, debug=True enabled in nova.conf:

root@stack-man:/var/lib/nova/instances# nova start 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances# nova show 84b14df7-7e29-49cc-bc9a-5f92ab050639
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| DemoNet1 network                     | 172.16.1.24                                              |
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | stack-srv1                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | stack-srv1                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000012                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-09-10T15:36:02.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2014-09-10T15:35:53Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 373a2ec65e77c9d55e76e6006cf40bf142fcffa471ff3dcfcff21d03 |
| id                                   | 84b14df7-7e29-49cc-bc9a-5f92ab050639                     |
| image                                | Attempt to boot from volume - no image supplied          |
| key_name                             | demo-key                                                 |
| metadata                             | {}                                                       |
| name                                 | Cirros1                                                  |
| os-extended-volumes:volumes_attached | [{"id": "5726609f-4e58-47bc-baf6-995a7cafdc1f"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | cb9ab00c01c44a85989d1886374e20e5                         |
| updated                              | 2014-09-10T15:53:37Z                                     |
| user_id                              | 4edeb1fc16dd415eb9225f492d2d4f59                         |
+--------------------------------------+----------------------------------------------------------+
root@stack-man:/var/lib/nova/instances# nova migrate 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances#

Then looking at /var/log/nova-compute.log on stack-srv2 shows the following:

2014-09-10 16:54:31.413 15923 DEBUG nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Stashing vm_state: active _prep_resize /usr/lib/python2.7/dist-packages/nova/compute/manager.py:3317
2014-09-10 16:54:31.562 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore "compute_resources" lock /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:168
2014-09-10 16:54:31.563 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore / lock "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:248
2014-09-10 16:54:31.563 15923 DEBUG nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Memory overhead for 512 MB instance; 0 MB resize_claim /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:159
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total memory: 3846 MB, used: 512.00 MB
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] memory limit: 5769.00 MB, free: 5257.00 MB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total disk: 297 GB, used: 0.00 GB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] disk limit not specified, defaulting to unlimited
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total CPUs: 4 VCPUs, used: 0.00 VCPUs
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] CPUs limit not specified, defaulting to unlimited
2014-09-10 16:54:31.566 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Claim successful
2014-09-10 16:54:31.643 15923 AUDIT nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Updating from migration 84b14df7-7e29-49cc-bc9a-5f92ab050639
2014-09-10 16:54:31.727 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Semaphore / lock released "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:252
2014-09-10 16:54:31.727 15923 AUDIT nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Migrating
2014-09-10 16:54:33.908 15923 ERROR nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Setting instance vm_state to ERROR
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Traceback (most recent call last):
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3581, in finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     disk_info, image)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3527, in _finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 259, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 391, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     instance=instance_p, migration=migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     wait_for_reply=True, timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     raise result
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] RemoteError: Remote error: Unauthorized {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1024, in network_migrate_instance_finish\n    migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 530, in network_migrate_instance_finish\n    self.network_api.migrate_instance_finish(context, instance, migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1013, in migrate_instance_finish\n    data = neutron.list_ports(**search_opts)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 101, in with_params\n    ret = self.function(instance, *args, **kwargs)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 308, in list_ports\n    **_params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1329, in list\n    for r in self._pagination(collection, path, **params):\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1342, in _pagination\n    res = self.get(path, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1315, in get\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1300, in retry_request\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1227, in do_request\n    self.httpclient.authenticate_and_fetch_endpoint_url()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 131, in authenticate_and_fetch_endpoint_url\n    self.authenticate()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 230, in authenticate\n    self._authenticate_keystone()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 208, in _authenticate_keystone\n    allow_redirects=True)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 117, in _cs_request\n    raise exceptions.Unauthorized(message=body)\n', u'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\n'].

An abbreviated error is then shown in the output of 'nova show'.

I must be missing something?

Migration between compute nodes fails

I'm running icehouse on Ubuntu 14.04 LTS, installed from the Ubuntu repositories.

I have the following nodes:

  • stack-man (controller)
  • stack-net (network)
  • stack-srv1,stack-srv2 (2 x compute)

Virtualisation plugin is KVM/Libvirt. Networking is via neutron using ml2 plugin with gre tunnels & openvswitch.

I have 3 networks - all nodes have an interface on the management network (their primary names, as above). The network and compute nodes each have an interface on the vm tunnel network (as configured and working in neutron, their hostnames are suffixed with -vm; stack-net-vm, stack-srv1-vm, etc.). Finally there is a storage traffic network for stack-man, stack-srv1/2 and the GlusterFS nodes (hostnames suffixed with -fs; stack-man-fs, stack-srv1-fs etc).

Migration and management commands are therefore only running over the management network.

Instance storage is provisioned by a GlusterFS mount on the controller and compute nodes (/var/lib/nova/instances is a link to the 'instance' brick from the GlusterFS nodes) provisioned by a pair of storage nodes. The storage nodes don't run any Openstack services.

Block storage is currently running on the controller node also - using the GlusterfsDriver in /etc/cinder/cinder.conf with the cinder volume configured in /etc/cinder/glusterfs.conf (using the 'cinder' brick from the GlusterFS nodes). At some point I'll move additional cinder-volume services to the compute nodes for resiliency.

Everything has been setup pretty much according to the install guide - I had some issues configuring libvirt to do remote access, but that all seems to work (certs all installed - virsh works from either the controller or compute nodes).

I've created an instance from the CirrOs image - it provisions a new volume, mounts it, creates an instance folder, starts the vm, I can login, I get DHCP assigned networking from neutron etc. All great. However, neither migrate nor live-migration features work.

/var/log/nova/nova-compute.log lists the following message regardless of whether migrate or live-migration is used:

'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\\n\'

All the usual command line interface tools work - nova list/stop/start/show, neutron agent-list, cinder list etc.

An actual trace of the sequence of actions for a non-live migration, debug=True enabled in nova.conf:

root@stack-man:/var/lib/nova/instances# nova start 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances# nova show 84b14df7-7e29-49cc-bc9a-5f92ab050639
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| DemoNet1 network                     | 172.16.1.24                                              |
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | stack-srv1                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | stack-srv1                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000012                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-09-10T15:36:02.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2014-09-10T15:35:53Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 373a2ec65e77c9d55e76e6006cf40bf142fcffa471ff3dcfcff21d03 |
| id                                   | 84b14df7-7e29-49cc-bc9a-5f92ab050639                     |
| image                                | Attempt to boot from volume - no image supplied          |
| key_name                             | demo-key                                                 |
| metadata                             | {}                                                       |
| name                                 | Cirros1                                                  |
| os-extended-volumes:volumes_attached | [{"id": "5726609f-4e58-47bc-baf6-995a7cafdc1f"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | cb9ab00c01c44a85989d1886374e20e5                         |
| updated                              | 2014-09-10T15:53:37Z                                     |
| user_id                              | 4edeb1fc16dd415eb9225f492d2d4f59                         |
+--------------------------------------+----------------------------------------------------------+
root@stack-man:/var/lib/nova/instances# nova migrate 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances#

Then looking at /var/log/nova-compute.log on stack-srv2 shows the following:

2014-09-10 16:54:31.413 15923 DEBUG nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Stashing vm_state: active _prep_resize /usr/lib/python2.7/dist-packages/nova/compute/manager.py:3317
2014-09-10 16:54:31.562 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore "compute_resources" lock /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:168
2014-09-10 16:54:31.563 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore / lock "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:248
2014-09-10 16:54:31.563 15923 DEBUG nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Memory overhead for 512 MB instance; 0 MB resize_claim /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:159
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total memory: 3846 MB, used: 512.00 MB
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] memory limit: 5769.00 MB, free: 5257.00 MB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total disk: 297 GB, used: 0.00 GB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] disk limit not specified, defaulting to unlimited
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total CPUs: 4 VCPUs, used: 0.00 VCPUs
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] CPUs limit not specified, defaulting to unlimited
2014-09-10 16:54:31.566 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Claim successful
2014-09-10 16:54:31.643 15923 AUDIT nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Updating from migration 84b14df7-7e29-49cc-bc9a-5f92ab050639
2014-09-10 16:54:31.727 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Semaphore / lock released "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:252
2014-09-10 16:54:31.727 15923 AUDIT nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Migrating
2014-09-10 16:54:33.908 15923 ERROR nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Setting instance vm_state to ERROR
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Traceback (most recent call last):
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3581, in finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     disk_info, image)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3527, in _finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 259, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 391, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     instance=instance_p, migration=migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     wait_for_reply=True, timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     raise result
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] RemoteError: Remote error: Unauthorized {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1024, in network_migrate_instance_finish\n    migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 530, in network_migrate_instance_finish\n    self.network_api.migrate_instance_finish(context, instance, migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1013, in migrate_instance_finish\n    data = neutron.list_ports(**search_opts)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 101, in with_params\n    ret = self.function(instance, *args, **kwargs)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 308, in list_ports\n    **_params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1329, in list\n    for r in self._pagination(collection, path, **params):\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1342, in _pagination\n    res = self.get(path, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1315, in get\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1300, in retry_request\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1227, in do_request\n    self.httpclient.authenticate_and_fetch_endpoint_url()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 131, in authenticate_and_fetch_endpoint_url\n    self.authenticate()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 230, in authenticate\n    self._authenticate_keystone()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 208, in _authenticate_keystone\n    allow_redirects=True)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 117, in _cs_request\n    raise exceptions.Unauthorized(message=body)\n', u'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\n'].

An abbreviated error is then shown in the output of 'nova show'.

The instance is still running on the original node at this point - I can 'nova reset-state ...' and then do stop/destroy etc all okay. Command line/networking etc are all still working.

I must be missing something?something obvious that is causing both live and non-live migration to fail?

Migration between compute nodes fails

I'm running icehouse on Ubuntu 14.04 LTS, installed from the Ubuntu repositories.

I have the following nodes:

  • stack-man (controller)
  • stack-net (network)
  • stack-srv1,stack-srv2 (2 x compute)

Virtualisation plugin is KVM/Libvirt. Networking is via neutron using ml2 plugin with gre tunnels & openvswitch.

I have 3 networks - all nodes have an interface on the management network (their primary names, as above). The network and compute nodes each have an interface on the vm tunnel network (as configured and working in neutron, their hostnames are suffixed with -vm; stack-net-vm, stack-srv1-vm, etc.). Finally there is a storage traffic network for stack-man, stack-srv1/2 and the GlusterFS nodes (hostnames suffixed with -fs; stack-man-fs, stack-srv1-fs etc).

Migration and management commands are therefore only running over the management network.

Instance storage is provisioned by a GlusterFS mount on the controller and compute nodes (/var/lib/nova/instances is a link to the 'instance' brick from the GlusterFS nodes) provisioned by a pair of storage nodes. The storage nodes don't run any Openstack services.

Block storage is currently running on the controller node also - using the GlusterfsDriver in /etc/cinder/cinder.conf with the cinder volume configured in /etc/cinder/glusterfs.conf (using the 'cinder' brick from the GlusterFS nodes). At some point I'll move additional cinder-volume services to the compute nodes for resiliency.

Everything has been setup pretty much according to the install guide - I had some issues configuring libvirt to do remote access, but that all seems to work (certs all installed - virsh works from either the controller or compute nodes).

I've created an instance from the CirrOs image - it provisions a new volume, mounts it, creates an instance folder, starts the vm, I can login, I get DHCP assigned networking from neutron etc. All great. However, neither migrate nor live-migration features work.

/var/log/nova/nova-compute.log lists the following message regardless of whether migrate or live-migration is used:

'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\\n\'

All the usual command line interface tools work - nova list/stop/start/show, neutron agent-list, cinder list etc.

An actual trace of the sequence of actions for a non-live migration, debug=True enabled in nova.conf:

root@stack-man:/var/lib/nova/instances# nova start 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances# nova show 84b14df7-7e29-49cc-bc9a-5f92ab050639
+--------------------------------------+----------------------------------------------------------+
| Property                             | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| DemoNet1 network                     | 172.16.1.24                                              |
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | stack-srv1                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | stack-srv1                                               |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000012                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2014-09-10T15:36:02.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2014-09-10T15:35:53Z                                     |
| flavor                               | m1.tiny (1)                                              |
| hostId                               | 373a2ec65e77c9d55e76e6006cf40bf142fcffa471ff3dcfcff21d03 |
| id                                   | 84b14df7-7e29-49cc-bc9a-5f92ab050639                     |
| image                                | Attempt to boot from volume - no image supplied          |
| key_name                             | demo-key                                                 |
| metadata                             | {}                                                       |
| name                                 | Cirros1                                                  |
| os-extended-volumes:volumes_attached | [{"id": "5726609f-4e58-47bc-baf6-995a7cafdc1f"}]         |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tenant_id                            | cb9ab00c01c44a85989d1886374e20e5                         |
| updated                              | 2014-09-10T15:53:37Z                                     |
| user_id                              | 4edeb1fc16dd415eb9225f492d2d4f59                         |
+--------------------------------------+----------------------------------------------------------+
root@stack-man:/var/lib/nova/instances# nova migrate 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances#

Then looking at /var/log/nova-compute.log on stack-srv2 shows the following:

2014-09-10 16:54:31.413 15923 DEBUG nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Stashing vm_state: active _prep_resize /usr/lib/python2.7/dist-packages/nova/compute/manager.py:3317
2014-09-10 16:54:31.562 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore "compute_resources" lock /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:168
2014-09-10 16:54:31.563 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Got semaphore / lock "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:248
2014-09-10 16:54:31.563 15923 DEBUG nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Memory overhead for 512 MB instance; 0 MB resize_claim /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:159
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total memory: 3846 MB, used: 512.00 MB
2014-09-10 16:54:31.564 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] memory limit: 5769.00 MB, free: 5257.00 MB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total disk: 297 GB, used: 0.00 GB
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] disk limit not specified, defaulting to unlimited
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Total CPUs: 4 VCPUs, used: 0.00 VCPUs
2014-09-10 16:54:31.565 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] CPUs limit not specified, defaulting to unlimited
2014-09-10 16:54:31.566 15923 AUDIT nova.compute.claims [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Claim successful
2014-09-10 16:54:31.643 15923 AUDIT nova.compute.resource_tracker [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Updating from migration 84b14df7-7e29-49cc-bc9a-5f92ab050639
2014-09-10 16:54:31.727 15923 DEBUG nova.openstack.common.lockutils [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] Semaphore / lock released "resize_claim" inner /usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py:252
2014-09-10 16:54:31.727 15923 AUDIT nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Migrating
2014-09-10 16:54:33.908 15923 ERROR nova.compute.manager [req-8edc9c43-2da2-4e22-9460-4af9874ec98b 4edeb1fc16dd415eb9225f492d2d4f59 cb9ab00c01c44a85989d1886374e20e5] [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Setting instance vm_state to ERROR
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] Traceback (most recent call last):
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3581, in finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     disk_info, image)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3527, in _finish_resize
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 259, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     migration)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 391, in network_migrate_instance_finish
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     instance=instance_p, migration=migration_p)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     wait_for_reply=True, timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     timeout=timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639]     raise result
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] RemoteError: Remote error: Unauthorized {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2014-09-10 16:54:33.908 15923 TRACE nova.compute.manager [instance: 84b14df7-7e29-49cc-bc9a-5f92ab050639] [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1024, in network_migrate_instance_finish\n    migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 530, in network_migrate_instance_finish\n    self.network_api.migrate_instance_finish(context, instance, migration)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1013, in migrate_instance_finish\n    data = neutron.list_ports(**search_opts)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 101, in with_params\n    ret = self.function(instance, *args, **kwargs)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 308, in list_ports\n    **_params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1329, in list\n    for r in self._pagination(collection, path, **params):\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1342, in _pagination\n    res = self.get(path, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1315, in get\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1300, in retry_request\n    headers=headers, params=params)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1227, in do_request\n    self.httpclient.authenticate_and_fetch_endpoint_url()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 131, in authenticate_and_fetch_endpoint_url\n    self.authenticate()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 230, in authenticate\n    self._authenticate_keystone()\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 208, in _authenticate_keystone\n    allow_redirects=True)\n', u'  File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 117, in _cs_request\n    raise exceptions.Unauthorized(message=body)\n', u'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\n'].

An abbreviated error is then shown in the output of 'nova show'.

The instance is still running on the original node at this point - I can 'nova reset-state ...' and then do stop/destroy etc all okay. Command line/networking etc are all still working.

I must be missing something obvious that is causing both live and non-live migration to fail?

(Edit) Live migration is already fully configured:

root@stack-srv1:~# grep -v ^# /etc/libvirt/libvirtd.conf | strings
listen_tls = 1
listen_tcp = 0
unix_sock_group = "libvirtd"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
auth_tls = "none"
tls_no_verify_certificate = 1
auth_tcp = "none"
root@stack-srv1:~# grep -v ^# /etc/default/libvirt-bin | strings
start_libvirtd="yes"
libvirtd_opts="-d -l"
root@stack-srv1:~# grep live /etc/nova/nova.conf 
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
live_migration_uri=qemu+tls://%s/system
root@stack-srv1:~#

Certificates (CA, server and client) are all installed and working for libvirt from either compute node without interactive prompts:

root@stack-srv1:~# virsh --connect qemu+tls://stack-srv2/system list
 Id    Name                           State
----------------------------------------------------
 4     instance-00000012              running

root@stack-srv1:~#

... and from the other node:

root@stack-srv2:~# virsh --connect qemu+tls://stack-srv1/system list
 Id    Name                           State
----------------------------------------------------

root@stack-srv2:~#