Migration between compute nodes fails
I'm running icehouse on Ubuntu 14.04 LTS, installed from the Ubuntu repositories.
I have the following nodes:
- stack-man (controller)
- stack-net (network)
- stack-srv1,stack-srv2 (2 x compute)
Virtualisation plugin is KVM/Libvirt. Networking is via neutron using ml2 plugin with gre tunnels & openvswitch.
I have 3 networks - all nodes have an interface on the management network (their primary names, as above). The network and compute nodes each have an interface on the vm tunnel network (as configured and working in neutron, their hostnames are suffixed with -vm; stack-net-vm, stack-srv1-vm, etc.). Finally there is a storage traffic network for stack-man, stack-srv1/2 and the GlusterFS nodes (hostnames suffixed with -fs; stack-man-fs, stack-srv1-fs etc).
Migration and management commands are therefore only running over the management network.
Instance storage is provisioned by a GlusterFS mount on the controller and compute nodes (/var/lib/nova/instances is a link to the 'instance' brick from the GlusterFS nodes) provisioned by a pair of storage nodes. The storage nodes don't run any Openstack services.
Block storage is currently running on the controller node also - using the GlusterfsDriver in /etc/cinder/cinder.conf with the cinder volume configured in /etc/cinder/glusterfs.conf (using the 'cinder' brick from the GlusterFS nodes). At some point I'll move additional cinder-volume services to the compute nodes for resiliency.
Everything has been setup pretty much according to the install guide - I had some issues configuring libvirt to do remote access, but that all seems to work (certs all installed - virsh works from either the controller or compute nodes).
I've created an instance from the CirrOs image - it provisions a new volume, mounts it, creates an instance folder, starts the vm, I can login, I get DHCP assigned networking from neutron etc. All great. However, neither migrate nor live-migration features work.
/var/log/nova/nova-compute.log lists the following message regardless of whether migrate or live-migration is used:
'Unauthorized: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}\\n\'
All the usual command line interface tools work - nova list/stop/start/show, neutron agent-list, cinder list etc.
An actual trace of the sequence of actions for a non-live migration, debug=True enabled in nova.conf:
root@stack-man:/var/lib/nova/instances# nova start 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances# nova show 84b14df7-7e29-49cc-bc9a-5f92ab050639
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| DemoNet1 network | 172.16.1.24 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | stack-srv1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | stack-srv1 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000012 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2014-09-10T15:36:02.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-09-10T15:35:53Z |
| flavor | m1.tiny (1) |
| hostId | 373a2ec65e77c9d55e76e6006cf40bf142fcffa471ff3dcfcff21d03 |
| id | 84b14df7-7e29-49cc-bc9a-5f92ab050639 |
| image | Attempt to boot from volume - no image supplied |
| key_name | demo-key |
| metadata | {} |
| name | Cirros1 |
| os-extended-volumes:volumes_attached | [{"id": "5726609f-4e58-47bc-baf6-995a7cafdc1f"}] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | cb9ab00c01c44a85989d1886374e20e5 |
| updated | 2014-09-10T15:53:37Z |
| user_id | 4edeb1fc16dd415eb9225f492d2d4f59 |
+--------------------------------------+----------------------------------------------------------+
root@stack-man:/var/lib/nova/instances# nova migrate 84b14df7-7e29-49cc-bc9a-5f92ab050639
root@stack-man:/var/lib/nova/instances#
Then looking at /var/log/nova-compute.log on stack-srv2 shows ...