Ask Your Question

libvirt broken cpuset on migration workarounds?

asked 2015-02-21 07:29:25 -0600

rgcs gravatar image


does anyone know how to deal with broken cpuset definition in openstack juno on migration? I have different host types, one type has 40 cpus, most of the rest only 32. If I have a VM running on the 40 cpu host and try to migrate it to the 32 cpu one, it almost always fails with an error in libvirt (on destination) showing:

error : virCgroupSetValueStr:684 : Invalid value '10-19,30-39' for 'cpuset.cpus': Invalid argument

which cannot work as there are not 40 cpus on the target host. If I migrate between 40 cpu hosts, everything works. If I reboot one 40 cpu host and disable hyperthreading, same issue occurs here too (since now it only has 20 "cpus"). It appears libvirt livemigration does take the source host's capabilities and not the destination when defining cpusets. Note that sometimes it works but I do not know how to trigger it and why.

Does anyone know a trick to force a correct cpuset on migration? It is important that I migrate those VMs away without stopping them. Things I tried that did not work:

  • edited the cpuset in virsh on source, then tried live migration. Still tries to set a cpuset > 32 cpus on target host with 32 cpus.
  • edited the cpuset and edited /var/lib/nova/instances/... libvirt.xml and tried live migration. Still tries to set a cpuset > 32 cpus on target host with 32 cpus.

Any help appreciated, including code hacks in nova/libvirt code to force a cpuset so that this works. Thanks!

edit retag flag offensive close merge delete

3 answers

Sort by ยป oldest newest most voted

answered 2015-03-02 17:19:13 -0600

osborng gravatar image

I have worked around this issue by changing the python code located in /usr/lib/python2.7/dist-packages/nova/virt/libvirt/ I just remarked out line 1613 like below. This stops the cpuset parameter from being added to the libvirt.xml file for the instances. I'm not sure what the purpose of the cpuset "feature" is. It appears to be some kind of CPU affinity setting, which I'm not sure is useful in a cloud architecture, but I digress. Hope this works for you.

vcpu.set("cpuset", hardware.format_cpu_spec(self.cpuset))

edit flag offensive delete link more

answered 2015-07-15 03:10:33 -0600

I think cpusets are used here as performance tuning, it restricts the set of CPUs the process will run on, to avoid trashing the caches of others CPUs in the system.

edit flag offensive delete link more

answered 2015-07-14 12:52:56 -0600

med gravatar image

It's not clear to me what else you are omitting if you completely comment out that vcpu.set command. This should be an upstream bug against nova. Checking...

edit flag offensive delete link more


It should be harmless to avoid the parameter, maybe loose a bit of performance...

vincent-legoll gravatar imagevincent-legoll ( 2015-07-15 05:18:51 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2015-02-21 07:29:25 -0600

Seen: 1,035 times

Last updated: Jul 15 '15