nova api takes 6s per GET /server per VM, after couple VM created dashboard is unusuable with 30s timeout per request.

asked 2020-07-28 06:28:55 -0500

Hi,

I'm on OpenStack Ussuri, fresh deployment using MAAS + Juju in 3 node cluster. It looks like each metadata API GET per VM takes 6s and after few VM created my dashboard is unusable as you get HTTP timeout, cli is working and --timings show exactly 6s per query and nova list is a calculation 6s* nr of VM. Don't know how to troubleshoot the API call to narrow down where the time is spent, so far I've only established the following:

nova list with 5VMs nova --timings list +--------------------------------------+--------------+--------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+------------------------------------------+ | 62206736-e9b4-4ddc-acf3-d10810b5627b | test-1 | ACTIVE | - | Running | priv=10.10.100.77 | | c2d09321-aa14-44a8-9ee6-341bba59b418 | test-2 | ACTIVE | - | Running | priv=10.10.100.62 | | 9b3068db-bd4d-4cc3-a6ff-38af45994b70 | test-ext-1 | ACTIVE | - | Running | priv=10.10.100.23; public01=10.10.50.201 | | fa59bc60-6dc2-461f-b4b8-ff1acab3a6ae | test-ext-2 | ACTIVE | - | Running | priv=10.10.100.44; public01=10.10.50.165 | | e42364ea-f044-41a4-98f1-7ed4de636ac6 | test-ext-big | ACTIVE | - | Running | priv=10.10.100.25; public01=10.10.50.157 | +--------------------------------------+--------------+--------+------------+-------------+------------------------------------------+ +---------------------------------------+-----------------------+ | url | seconds | +---------------------------------------+-----------------------+ | auth_url https://10.10.40.203:5000/v3 | 0.0005016326904296875 | | GET /servers/detail | 30.854870557785034 | | Total | 30.855372190475464 | +---------------------------------------+-----------------------+

Per VM query

nova --timings stop test-ext-1 Request to stop server test-ext-1 has been accepted. +-----------------------------------------------------------+-----------------------+ | url | seconds | +-----------------------------------------------------------+-----------------------+ | auth_url https://10.10.40.203:5000/v3 | 0.0005033016204833984 | | GET /servers?name=test-ext-1 | 0.24817657470703125 | | GET /servers/9b3068db-bd4d-4cc3-a6ff-38af45994b70 | 6.564579248428345 | | POST /servers/9b3068db-bd4d-4cc3-a6ff-38af45994b70/action | 0.22148728370666504 | | Total | 7.034746408462524 | +-----------------------------------------------------------+-----------------------+

Time after I've removed three VMs

nova --timings list +--------------------------------------+------------+---------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------+---------+------------+-------------+------------------------------------------+ | 62206736-e9b4-4ddc-acf3-d10810b5627b | test-1 | SHUTOFF | - | Shutdown | priv=10.10.100.77 | | 9b3068db-bd4d-4cc3-a6ff-38af45994b70 | test-ext-1 | SHUTOFF | - | Shutdown | priv=10.10.100.23; public01=10.10.50.201 | +--------------------------------------+------------+---------+------------+-------------+------------------------------------------+ +---------------------------------------+-----------------------+ | url | seconds | +---------------------------------------+-----------------------+ | auth_url https://10.10.40.203:5000/v3 | 0.0004799365997314453 | | GET /servers/detail | 12.686200618743896 | | Total | 12.686680555343628 | +---------------------------------------+-----------------------+

edit retag flag offensive close merge delete