Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

live migration fails with nfs4 mounted /var/lib/nova/instances

I'm attempting to set up live migration of instances, I have two nodes in my pod, both running ubuntu 11.04, and using http://ppa.launchpad.net/nova-core/trunk/ubuntu ppa to install nova. They have a public 10.4.78.0/24 network attached to eth0 and a 192.168.0.0/24 private network. I'm using glance for image storage. The master node with glance, nova-api, nova-compute, nova-network and nova-volume on it is sharing /var/lib/nova/instances with the compute node via nfs4. Both nodes have kvm virtualization enabled.

I can start instances on the master node, but not the compute node and I can not live migrate from the master to the compute. When I run

nova-manage vm live_migrate i-00000016 csvirt-2

I get the following errors in csvirt-2's /var/log/nova/nova-compute.log

2011-07-13 10:58:42,549 DEBUG nova.compute.manager [-] instance network_info: |[[{u'injected': False, u'bridge': u'br_vlan1', u'cidr_v6': None, u'cidr': u'172.16.1.0/24', u'id': 1}, {u'label': u'vlan1', u'broadcast': u'172.16.1.255', u'ips': [{u'ip': u'172.16.1.9', u'netmask': u'255.255.255.0', u'enabled': u'1'}], u'mac': u'02:16:3e:62:f0:91', u'rxtx_cap': 0, u'dns': [None], u'gateway': u'172.16.1.7'}]]| from (pid=1037) _run_instance /usr/lib/pymodules/python2.7/nova/compute/manager.py:295 2011-07-13 10:58:42,553 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_vlan" for method "ensure_vlan"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:42,553 DEBUG nova.utils [-] Attempting to grab file lock "ensure_vlan" for method "ensure_vlan"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:42,554 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev vlan1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,558 DEBUG nova.utils [-] Result was 255 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:161 2011-07-13 10:58:42,558 DEBUG nova.linux_net [-] Starting VLAN inteface vlan1 from (pid=1037) ensure_vlan /usr/lib/pymodules/python2.7/nova/network/linux_net.py:465 2011-07-13 10:58:42,559 DEBUG nova.utils [-] Running cmd (subprocess): sudo vconfig set_name_type VLAN_PLUS_VID_NO_PAD from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,713 DEBUG nova.utils [-] Running cmd (subprocess): sudo vconfig add eth1 1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,722 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip link set vlan1 up from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,733 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_bridge" for method "ensure_bridge"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:42,733 DEBUG nova.utils [-] Attempting to grab file lock "ensure_bridge" for method "ensure_bridge"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:42,733 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev br_vlan1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,737 DEBUG nova.utils [-] Result was 255 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:161 2011-07-13 10:58:42,738 DEBUG nova.linux_net [-] Starting Bridge interface for vlan1 from (pid=1037) ensure_bridge /usr/lib/pymodules/python2.7/nova/network/linux_net.py:488 2011-07-13 10:58:42,738 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl addbr br_vlan1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,747 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl setfd br_vlan1 0 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,756 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl stp br_vlan1 off from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,766 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip link set br_vlan1 up from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,776 DEBUG nova.utils [-] Running cmd (subprocess): sudo route -n from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,785 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip addr show dev vlan1 scope global from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,804 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl addif br_vlan1 vlan1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,062 DEBUG nova.virt.libvirt_conn [-] instance instance-00000018: starting toXML method from (pid=1037) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:1035 2011-07-13 10:58:43,118 DEBUG nova.virt.libvirt_conn [-] instance instance-00000018: finished toXML method from (pid=1037) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:1039 2011-07-13 10:58:43,119 INFO nova [-] called setup_basic_filtering in nwfilter 2011-07-13 10:58:43,119 INFO nova [-] ensuring static filters 2011-07-13 10:58:43,149 DEBUG nova.virt.libvirt.firewall [-] iptables firewall: Setup Basic Filtering from (pid=1037) setup_basic_filtering /usr/lib/pymodules/python2.7/nova/virt/libvirt/firewall.py:537 2011-07-13 10:58:43,150 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:43,150 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:43,152 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:43,152 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:43,153 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,163 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,174 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,183 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,206 DEBUG nova.virt.libvirt.firewall [-] Adding security group rule: <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3426350=""> from (pid=1037) instance_rules /usr/lib/pymodules/python2.7/nova/virt/libvirt/firewall.py:663 2011-07-13 10:58:43,206 DEBUG nova.virt.libvirt.firewall [-] Adding security group rule: <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x34263d0=""> from (pid=1037) instance_rules /usr/lib/pymodules/python2.7/nova/virt/libvirt/firewall.py:663 2011-07-13 10:58:43,206 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:43,207 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:43,207 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,217 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,227 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,237 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,247 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000018/ from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,266 INFO nova.virt.libvirt_conn [-] instance instance-00000018: Creating image 2011-07-13 10:58:43,402 DEBUG nova.utils [-] Attempting to grab semaphore "00000001" for method "call_if_not_exists"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:43,403 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/00000001 /var/lib/nova/instances/instance-00000018/kernel from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:43,705 DEBUG nova.utils [-] Attempting to grab semaphore "da4b9237bacccdf19c0760cab7aec4a8359010b0" for method "call_if_not_exists"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:43,706 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/da4b9237bacccdf19c0760cab7aec4a8359010b0 /var/lib/nova/instances/instance-00000018/disk from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:44,008 DEBUG nova.utils [-] Attempting to grab semaphore "local_20" for method "call_if_not_exists"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:44,009 DEBUG nova.utils [-] Running cmd (subprocess): qemu-img create -f qcow2 -o cluster_size=2M,backing_file=/var/lib/nova/instances/_base/local_20 /var/lib/nova/instances/instance-00000018/disk.local from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:44,306 INFO nova.virt.libvirt_conn [-] instance instance-00000018: injecting key into image 2 2011-07-13 10:58:44,307 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -c /dev/nbd15 /var/lib/nova/instances/instance-00000018/disk from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:45,341 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/nbd15 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,734 DEBUG nova.utils [-] Running cmd (subprocess): sudo mount /dev/nbd15 /tmp/tmpVtG4uX from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,802 DEBUG nova.utils [-] Running cmd (subprocess): sudo mkdir -p /tmp/tmpVtG4uX/root/.ssh from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,813 DEBUG nova.utils [-] Running cmd (subprocess): sudo chown root /tmp/tmpVtG4uX/root/.ssh from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,823 DEBUG nova.utils [-] Running cmd (subprocess): sudo chmod 700 /tmp/tmpVtG4uX/root/.ssh from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,832 DEBUG nova.utils [-] Running cmd (subprocess): sudo tee -a /tmp/tmpVtG4uX/root/.ssh/authorized_keys from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:48,852 DEBUG nova.utils [-] Running cmd (subprocess): sudo umount /dev/nbd15 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:52,224 DEBUG nova.utils [-] Running cmd (subprocess): rmdir /tmp/tmpVtG4uX from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:52,237 DEBUG nova.utils [-] Running cmd (subprocess): sudo qemu-nbd -d /dev/nbd15 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:55,062 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 87, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 591, in spawn (nova.exception): TRACE: domain = self._create_new_domain(xml) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 1087, in _create_new_domain (nova.exception): TRACE: domain.createWithFlags(launch_flags) (nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 337, in createWithFlags (nova.exception): TRACE: if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) (nova.exception): TRACE: libvirtError: internal error Process exited while reading console log output: chardev: opening backend "file" failed (nova.exception): TRACE: (nova.exception): TRACE: 2011-07-13 10:58:55,088 ERROR nova.compute.manager [-] Instance '24' failed to spawn. Is virtualization enabled in the BIOS? Details: internal error Process exited while reading console log output: chardev: opening backend "file" failed (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 311, in _run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance, network_info, bd_mapping) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 93, in _wrap (nova.compute.manager): TRACE: raise Error(str(e)) (nova.compute.manager): TRACE: Error: internal error Process exited while reading console log output: chardev: opening backend "file" failed (nova.compute.manager): TRACE: (nova.compute.manager): TRACE:

Here is a copy of the nova.conf that the nodes share

RabbitMQ

--rabbit_host=10.4.78.190

MySQL

--sql_connection=mysql://nova:nova@10.4.78.190/nova

Networking

--network_manager=nova.network.manager.VlanManager --vlan_interface=eth1 --public_interface=eth0 --network_host=10.4.78.190 --routing_source_ip=10.4.78.190 --fixed_range=192.168.0.0/24 --network_size=255 --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge

Virtualization

--libvirt_type=kvm

Volumes

--iscsi_ip_prefix=10.4.78.190

--num_targets=100

APIs

--auth_driver=nova.auth.dbdriver.DbDriver --cc_host=10.4.78.190 --ec2_url=http://10.4.78.190:8773/services/Cloud --s3_host=10.4.78.190 --s3_dmz=10.4.78.190

Image service

--glance_api_servers=10.4.78.190:9292 --image_service=nova.image.glance.GlanceImageService

Misc

--logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --verbose

VNC Console

--vnc_enabled=true --vncproxy_url=http://10.4.78.190:6080 --vnc_console_proxy_url=http://10.4.78.190:6080