libvirt broken cpuset on migration workarounds?
Hello,
does anyone know how to deal with broken cpuset definition in openstack juno on migration? I have different host types, one type has 40 cpus, most of the rest only 32. If I have a VM running on the 40 cpu host and try to migrate it to the 32 cpu one, it almost always fails with an error in libvirt (on destination) showing:
error : virCgroupSetValueStr:684 : Invalid value '10-19,30-39' for 'cpuset.cpus': Invalid argument
which cannot work as there are not 40 cpus on the target host. If I migrate between 40 cpu hosts, everything works. If I reboot one 40 cpu host and disable hyperthreading, same issue occurs here too (since now it only has 20 "cpus"). It appears libvirt livemigration does take the source host's capabilities and not the destination when defining cpusets. Note that sometimes it works but I do not know how to trigger it and why.
Does anyone know a trick to force a correct cpuset on migration? It is important that I migrate those VMs away without stopping them. Things I tried that did not work:
- edited the cpuset in virsh on source, then tried live migration. Still tries to set a cpuset > 32 cpus on target host with 32 cpus.
- edited the cpuset and edited /var/lib/nova/instances/... libvirt.xml and tried live migration. Still tries to set a cpuset > 32 cpus on target host with 32 cpus.
Any help appreciated, including code hacks in nova/libvirt code to force a cpuset so that this works. Thanks!
