Ask Your Question
0

live migration fails with nfs4 mounted /var/lib/nova/instances

asked 2011-07-13 17:21:41 -0500

p-spencer-davis gravatar image

I'm attempting to set up live migration of instances, I have two nodes in my pod, both running ubuntu 11.04, and using http://ppa.launchpad.net/nova-core/trunk/ubuntu (http://ppa.launchpad.net/nova-core/tr...) ppa to install nova. They have a public 10.4.78.0/24 network attached to eth0 and a 192.168.0.0/24 private network. I'm using glance for image storage. The master node with glance, nova-api, nova-compute, nova-network and nova-volume on it is sharing /var/lib/nova/instances with the compute node via nfs4. Both nodes have kvm virtualization enabled.

I can start instances on the master node, but not the compute node and I can not live migrate from the master to the compute. When I run

nova-manage vm live_migrate i-00000016 csvirt-2

I get the following errors in csvirt-2's /var/log/nova/nova-compute.log

2011-07-13 10:58:42,549 DEBUG nova.compute.manager [-] instance network_info: |[[{u'injected': False, u'bridge': u'br_vlan1', u'cidr_v6': None, u'cidr': u'172.16.1.0/24', u'id': 1}, {u'label': u'vlan1', u'broadcast': u'172.16.1.255', u'ips': [{u'ip': u'172.16.1.9', u'netmask': u'255.255.255.0', u'enabled': u'1'}], u'mac': u'02:16:3e:62:f0:91', u'rxtx_cap': 0, u'dns': [None], u'gateway': u'172.16.1.7'}]]| from (pid=1037) _run_instance /usr/lib/pymodules/python2.7/nova/compute/manager.py:295 2011-07-13 10:58:42,553 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_vlan" for method "ensure_vlan"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 10:58:42,553 DEBUG nova.utils [-] Attempting to grab file lock "ensure_vlan" for method "ensure_vlan"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:605 2011-07-13 10:58:42,554 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev vlan1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,558 DEBUG nova.utils [-] Result was 255 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:161 2011-07-13 10:58:42,558 DEBUG nova.linux_net [-] Starting VLAN inteface vlan1 from (pid=1037) ensure_vlan /usr/lib/pymodules/python2.7/nova/network/linux_net.py:465 2011-07-13 10:58:42,559 DEBUG nova.utils [-] Running cmd (subprocess): sudo vconfig set_name_type VLAN_PLUS_VID_NO_PAD from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,713 DEBUG nova.utils [-] Running cmd (subprocess): sudo vconfig add eth1 1 from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,722 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip link set vlan1 up from (pid=1037) execute /usr/lib/pymodules/python2.7/nova/utils.py:143 2011-07-13 10:58:42,733 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_bridge" for method "ensure_bridge"... from (pid=1037) inner /usr/lib/pymodules/python2.7/nova/utils.py:600 2011-07-13 ... (more)

edit retag flag offensive close merge delete

13 answers

Sort by » oldest newest most voted
0

answered 2012-04-03 17:59:52 -0500

vishvananda gravatar image

two questions:

1) is it possible to chown files on your nfs mount?

2) check syslog for AppArmor issues. You may have to create some special rules for AppArmor or disable apparmor in /etc/libvirt/qemu.conf

edit flag offensive delete link more
0

answered 2012-04-04 11:01:46 -0500

mandarvaze gravatar image

Vish :

1) is it possible to chown files on your nfs mount?

Not sure whom should I give the ownership to ? As the dir listing shows above, the files are created with correct owner/group settings. These files were created by nova-compute process.

I "touch"ed a file for testing - which was created with "mandar:mandar" - I tried changing the ownership, but I keep getting "invalid" argument - not sure if that is what you were referring to ..

mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ touch x mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ ll total 16 drwxrwxrwx 4 root root 4096 2012-04-04 03:46 ./ drwxr-xr-x 15 mandar mandar 4096 2012-04-04 03:39 ../ drwxrwxr-x 2 mandar libvirtd 4096 2012-04-04 02:18 _base/ drwxr-xr-x 2 mandar libvirtd 4096 2012-04-04 03:41 instance-0000000f/ -rw-rw-r-- 1 mandar mandar 0 2012-04-04 03:46 x mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ chown root x chown: changing ownership of x': Invalid argument mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ ll total 16 drwxrwxrwx 4 root root 4096 2012-04-04 03:46 ./ drwxr-xr-x 15 mandar mandar 4096 2012-04-04 03:39 ../ drwxrwxr-x 2 mandar libvirtd 4096 2012-04-04 02:18 _base/ drwxr-xr-x 2 mandar libvirtd 4096 2012-04-04 03:41 instance-0000000f/ -rw-rw-r-- 1 mandar mandar 0 2012-04-04 03:46 x mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ sudo chown root x chown: changing ownership ofx': Invalid argument mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ chgrp libvirtd x chgrp: changing group of x': Invalid argument mandar@ubuntu-dev-mandar:~/nfs_shared_instances_path$ sudo chgrp libvirtd x chgrp: changing group ofx': Invalid argument

I also debugged the "_chown_console_log_for_instance" in "nova/virt/libvirt/connection.py" and with breakpoint before "if os.path.exists(console_log)" - I touched "console.log" at the appropriate path from another terminal.

Over there also I am getting the same error:

2012-04-04 03:56:55 TRACE nova.rpc.amqp Command: sudo /usr/local/bin/nova-rootwrap chown 1000 /home/mandar/nfs_shared_instances_path/instance-00000010/console.log 2012-04-04 03:56:55 TRACE nova.rpc.amqp Exit code: 1 2012-04-04 03:56:55 TRACE nova.rpc.amqp Stdout: '' 2012-04-04 03:56:55 TRACE nova.rpc.amqp Stderr: "/bin/chown: changing ownership of `/home/mandar/nfs_shared_instances_path/instance-00000010/console.log': Invalid argument\n"

So what is the fix for this ?

edit flag offensive delete link more
0

answered 2012-04-04 12:49:13 -0500

mandarvaze gravatar image

Turns out there is some problem with nfs v4 – So we mounted the share using nfs version 3 (mount –o vers=3)

Now we could “chown” the files in nfs folder, and the instance creation also worked !!!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2011-07-13 17:21:41 -0500

Seen: 745 times

Last updated: Jun 19 '12