Ask Your Question
1

why shared cpu policy schedule all vcpu on same host cpu?

asked 2017-02-16 08:25:57 -0600

Maple gravatar image

I'm using extra specs of flavor to assign host resource to my VM on openstack mitaka, it works well when I want to pin cpus to my VM by using "hw:cpu_policy=dedicated" in flavor, but what shocks me is that if I don't pin cpus or use "hw:cpu_policy=shared", all vcpus always use or be scheduled on the same host cpu even though its cpu-set has other cpus to choose, for exmaple:

virsh vcpuinfo instance-000000df

VCPU: 0 CPU: 3 State: running CPU time: 2.9s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 1 CPU: 3 State: running CPU time: 0.3s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 2 CPU: 3 State: running CPU time: 0.5s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 3 CPU: 3 State: running CPU time: 0.2s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 4 CPU: 3 State: running CPU time: 0.3s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 5 CPU: 3 State: running CPU time: 0.2s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 6 CPU: 3 State: running CPU time: 0.3s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

VCPU: 7 CPU: 3 State: running CPU time: 0.2s CPU Affinity: ---yyyyyyy-------------yyyyyyy----------

you can see that all vcpus has CPU affinity which has more than one CPU, but they are all scheduled on the same and first CPU of CPU affinity group, the corresponding xml of libevirt is:

<vcpu placement="static">8</vcpu> <cputune> <vcpupin vcpu="0" cpuset="3-9,23-29"/> <vcpupin vcpu="1" cpuset="3-9,23-29"/> <vcpupin vcpu="2" cpuset="3-9,23-29"/> <vcpupin vcpu="3" cpuset="3-9,23-29"/> <vcpupin vcpu="4" cpuset="3-9,23-29"/> <vcpupin vcpu="5" cpuset="3-9,23-29"/> <vcpupin vcpu="6" cpuset="3-9,23-29"/> <vcpupin vcpu="7" cpuset="3-9,23-29"/> <emulatorpin cpuset="3-9,23-29"/> </cputune>

is this expected behavior? But I saw the openstack official guide introduce that hw:cpu_policy=shared will make vcpus freely floating across cpu-set. Or I'm not using it right?

best regards.

edit retag flag offensive close merge delete
add a comment

3 answers

Sort by ยป oldest newest most voted
1

answered 2017-02-16 09:29:12 -0600

volenbovsky gravatar image

Hi, I think you have those CPUs in 'isolcpus' in kernel command line. And Linux scheduler doesn't load share between those. See https://bugzilla.redhat.com/show_bug....

edit flag offensive delete link more

Comments

1

Hi, great thanks. it's exactly what I want to know. But according to the post you gave, if I don't use 'isolcpus' and only use 'vcpu_pin_set' in nova, that can ensure host don't use cpu set reserved by nova? This is what I'm concerning since I use isolcpus for this reason after all.

Maple gravatar imageMaple ( 2017-02-16 20:44:22 -0600 )edit
1

Steve Gordon pointed to CPUAffinity above

volenbovsky gravatar imagevolenbovsky ( 2017-02-17 13:13:46 -0600 )edit

Yeah, I also went back and added a bit more detail relating back to this specific example.

sgordon gravatar imagesgordon ( 2017-02-17 13:23:49 -0600 )edit
add a comment
1

answered 2017-02-17 08:03:27 -0600

sgordon gravatar image

updated 2017-02-17 13:25:38 -0600

Assuming you aren't running real-time guests, use systemd's CPUAffinity setting instead of isolcpus. It works the opposite way to isolcpus though instead of specifying the cores where you don't want host processes to run, you specify the cores where you do want them to run. You still need to use vcpu_pin_set in the same way as before.

There's not a heap of documentation out there on using the CPUAffinity setting but you can find it in the systemd configuration man page:

https://www.freedesktop.org/software/...

Relating to your example if I want to force host processes to 0 and 1:

CPUAffinity=0 1

Then in my /etc/nova/nova.conf I tell it to put the guests on the other cores:

vcpu_pin_set=2-7

Take note that unlike the isolcpus case I have to use different ranges for the two settings as they in effect mean the inverse of each other.

Unfortunately CPUAffinity does not restrict kernel threads themselves, but it does avoid some of the real-time focused side-effects of isolcpus and is the best way of handling this for now as far as I am aware.

edit flag offensive delete link more
add a comment
0

answered 2017-02-19 04:32:53 -0600

Maple gravatar image

Hi Steve,

Thanks very much for you recommendation. Unfortunately, systemd is not used in my environment, but it will be good option when we use it. The problem actually is caused by cpus in isolcpus won't be scheduled by kernel under default schedule algorithm, so we decide to use chrt as workaround to change schedule algorithm for VM process which want to float on range of cpus, here is chrt ref: http://man7.org/linux/man-pages/man1/....

best regards.

edit flag offensive delete link more
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2017-02-16 07:19:37 -0600

Seen: 1,165 times

Last updated: Feb 19 '17

Related questions

Issue with openstack flavor set

CPU topology and NUMA topology

Getting Nova Server From Metadata

Auto update hostnames in DNS

nova-conductor usage cpu 90%

Scalr Openstack flavor error

Maxing only Single core on instance with multiple vcpus

Deleted instance from compute (VMWARE ESX)

instance launched with flavour doesn't add required Hard disk

Instance failed network setup after 1 attempt(s): PortBindingFailed: Binding failed for port 9c3fdd10-e1c1-4280-93df-9b1079a3ab41, please check neutron logs for more information.