Ask Your Question
0

nova scheduler - no allocation candidates from the placement API

asked 2018-08-12 00:23:18 -0500

kubatyszko gravatar image

Hi there, I have a fresh setup of Queens with 2 compute nodes. The instances schedule across the nodes just fine, including GPU passthrough (both compute nodes have the same GPU and configuration). As long as a GPU server has 4GB RAM or less, it works, but as soon as I create new flavor with more RAM, the scheduler fails with this message:

2018-08-11 21:25:34.798 6832 DEBUG nova.scheduler.manager [req-5adf421b-dd4b-4a41-a09d-640d6bd98e9c 1cbff9eecbc340799a1a40a1b8875a39 1e99608321574fe8bb1e55a95319d6f5 - default default] Starting to schedule for instances: [u'2a22fc14-a028-4a3b-a14a-2abc270bd55d'] select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/manager.py:110 2018-08-11 21:25:36.017 6832 DEBUG nova.scheduler.manager [req-5adf421b-dd4b-4a41-a09d-640d6bd98e9c 1cbff9eecbc340799a1a40a1b8875a39 1e99608321574fe8bb1e55a95319d6f5 - default default] Got no allocation candidates from the Placement API. This may be a temporary occurrence as compute nodes start up and begin reporting inventory to the Placement service. select_destinations /usr/lib/python2.7/site-packages/nova/scheduler/manager.py:135

When calling API manually, I AM getting response without issues:

curl -g -i -X GET http://192.168.44.20:8778/allocation_candidates?resources=DISK_GB:1,MEMORY_MB:64000,VCPU:1 (http://192.168.44.20:8778/allocation_...) \ -H "User-Agent: python-novaclient" \ -H "Accept: application/json" \ -H "X-Auth-Token: gAAAAABbb8Ixv_x6uNJcy0S2Uyma6vAo6tXjJXTXUXqH2J5A-eggGTBKpXqdxbNf-cwDFt44Dxh9JXEH65ai7ILPEmPKHrQMKCUK_LjF0aLPMZbb4BIPn5S1Y_4HQ2vNvf9gIFANMkLVj4gW8zUD9fjGemCEiub7jZs2ZdpowHqDcNFlZI6Zt_Q" \ -H "OpenStack-API-Version: placement 1.10"

{"provider_summaries": {"467507a7-b105-4b07-8153-297cfd584181": {"resources": {"VCPU": {"used": 0, "capacity": 384}, "MEMORY_MB": {"used": 0, "capacity": 294139}, "DISK_GB": {"used": 0, "capacity": 49}}}, "95f20421-1c99-49b2-9954-9d13a32df9a8": {"resources": {"VCPU": {"used": 8, "capacity": 384}, "MEMORY_MB": {"used": 4096, "capacity": 294139}, "DISK_GB": {"used": 30, "capacity": 49}}}}, "allocation_requests": [{"allocations": [{"resource_provider": {"uuid": "95f20421-1c99-49b2-9954-9d13a32df9a8"}, "resources": {"VCPU": 1, "MEMORY_MB": 64000, "DISK_GB": 1}}]}, {"allocations": [{"resource_provider": {"uuid": "467507a7-b105-4b07-8153-297cfd584181"}, "resources": {"VCPU": 1, "MEMORY_MB": 64000, "DISK_GB": 1}}]}]}

Scheduling instances without GPU assigned, I was able to get maximum 9GB RAM allocated, anything more fails with the same error.

I'm not sure how to debug this, since placement service doesn't have its own logging.

edit retag flag offensive close merge delete

Comments

The placement service does have logging but it seems to end up in different places depending on how the service is deployed.

Or are you saying that the logging in there but it is not complete enough? If the latter, improvements to that have been backported to rocky, but I'm not sure about queens.

cdent gravatar imagecdent ( 2018-12-20 06:37:01 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-08-13 15:50:00 -0500

kubatyszko gravatar image

I solved it. It turns out that the flavor's default disk size was set to 100GB and my compute nodes don't have enough.

What's strange is that even though I'm using an external volume over iscsi (already provisioneg), the placement still asks the compute nodes for 100GB storage anyway - Hence "no candidates".

What's even more strange, is that the allocation works just fine with flavors requesting 100GB disk but less RAM...

edit flag offensive delete link more

Comments

If your flavor is asking for disk, then placement will be asked for disk. It sounds like your flavors may not be ideally set up.

cdent gravatar imagecdent ( 2018-12-20 06:38:34 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2018-08-11 23:35:26 -0500

Seen: 279 times

Last updated: Aug 12 '18