devstack queens : multiple uwsgi processes consuming CPU

asked 2018-07-17 11:01:40 -0500

anonymous user

Anonymous

Hi I have installed devstack stable/queens on an ubuntu16.04 VM (KVM 4vCPU, 10GB RAM, 30G disk) on dell xps15 laptop (ubuntu18.04 host) (+heat/tacker/mistral/barbican/networking-sfc which tacker requires).

As this is a laptop I often put it in suspend mode (note suspend, not hibernate) , which is probably unusual for Devstack. In any case it seemed (to date) to come out of suspend correctly and Openstack Horizon/CLI was accessible etc. each time over past week. I have basically got one Instance running using cirros image (via Tacker vnfd-create and vnf-create examples)

However, today , I now seem to have 20 uwsgi processes running when checking with Top, each using 10-35% CPU, can't use Horizon or openstack-python client on Host any more. Is this normal? (the number of uwsgi processes , I understand many services now use uwsgi)

I tried a

sudo systemctl restart devstack@*

(on the VM of course)

Job for devstack@g-api.service failed because the control process exited with error code. See "systemctl status devstack@g-api.service" and "journalctl -xe" for details.

systemctl status devstack@g-api.service
● devstack@g-api.service - Devstack devstack@g-api.service
   Loaded: loaded (/etc/systemd/system/devstack@g-api.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2018-07-17 14:00:35 UTC; 1h 28min ago
  Process: 904 ExecStart=/usr/local/bin/uwsgi --procname-prefix glance-api --ini /etc/glance/glance-u
 Main PID: 904 (code=exited, status=1/FAILURE)
   Status: "initializing uWSGI"
   CGroup: /system.slice/system-devstack.slice/devstack@g-api.service
           ├─24615 glance-apiuWSGI worker 
           └─24616 glance-apiuWSGI worker

This restart didn't kill the uwsgi processes

Is there any solution to gracefully recover ?

Of course I could unstack.sh and stack.sh but will lose all created instances/networks etc

I resigned my self to finally do a

killall -SIGKILL uwsgi

which killed all the uwsgi processes (...and the laptop fan finally stopped...)

After this could reconnect via Horizon or CLI (and it seemed my images/instances/networks still intact), can ssh into my Instance still.

Any idea why so many uwsgi processes appear? What do I disable by killing all uwsgi ? (my devstack still seems to be working ok apparently)

thanks

edit retag flag offensive close merge delete

Comments

When I hear “suspend” I wonder if this is related to timing? Are there activities that get tripped by a sudden jump in the system time? RabbitMQ is very time-sensitive, for example.

If g-api doesn’t start, check its journal journalctl -u devstack@g-api.

Bernd Bausch gravatar imageBernd Bausch ( 2018-07-17 16:47:48 -0500 )edit

most probably, interestingly in another installation using Packstack-all-in-one, its beam.smp (rabbitmq) that consumes CPU after resuming from suspend, similar timing issues I think .

After killing uwsgi above, I needed a 'sudo systemctl restart devstack@g-api.service' to see my Images again.

AndyW gravatar imageAndyW ( 2018-07-18 09:35:55 -0500 )edit