Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Can't spawn instances due to no ram

I was able to install openstack on centos using packstack (this manual) (I have one machine that run all openstack modules - hypervisor and compute).

I was able to spawn ubuntu16.04 cloud image once, with flavor m1.small.
I had to delete it and create a new instance because of wrong key pair.
Since then, I couldn't spawn any instance.

I was able to see in nova-compute.log the next error message:

qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory

free -h

          total        used        free      shared  buff/cache   available
Mem:            15G         14G        222M         54M        356M        120M
Swap:          6.8G        6.8G         20K

I don't understand where all the RAM went.

This is what top show (no one use that much RAM).

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                 
733 apache    20   0 1495388 253644   2876 S   5.6  1.6   3:02.59 httpd                                                                                                   
2129 mysql     20   0 9079668 250452   3616 S  27.8  1.5 211:05.47 mysqld                                                                                                  
734 apache    20   0 1371060 171732   2328 S   0.0  1.1   1:02.65 httpd                                                                                                   
23638 nova      20   0  513640 163888   1632 S   0.0  1.0   0:21.08 nova-api                                                                                                
23633 nova      20   0  512032 163560   2012 R  88.9  1.0  44:30.40 nova-api                                                  
...

I don't think that's the case of linux use the memory for io cache (as described here) because you can see the buff/cache is only 356M.

On the other hand, this is what openstack hypervisor show 1 -f json show:

...
  "memory_mb_used": 16016, 
  "uptime": "5 days,  3:05", 
  "host_time": "08:02:38", 
  "local_gb_used": 12, 
  "id": 1, 
  "current_workload": 0, 
  "state": "up", 
  "load_average": "5.01, 9.39, 6.09", 
  "users": "4", 
  "aggregates": [], 
  "host_ip": "192.168.201.209", 
  "hypervisor_hostname": "cloud.centos.lan", 
  "hypervisor_version": 2010000, 
  "disk_available_least": 27, 
  "local_gb": 40, 
  "free_ram_mb": 15861, 
  "vcpus_used": 0, 
  "hypervisor_type": "QEMU", 
  "memory_mb": 16373, 
  "vcpus": 16, 
  "running_vms": 0, 
  "service_id": 9, 
  "service_host": "cloud.centos.lan"
  • Is there something I can do in order to start instances?
  • What is taking all this RAM?
  • I tried flushing Linux io cache without any help

Why there are so many processes that take all of my physical RAM ?

 ps -e -o pid,vsz,rss,comm= | sort -n -k 3  
 ...  
 7999 623252 96480 httpd  
 8019 623252 96648 httpd  
 8068 623252 96824 httpd  
 7920 623500 96960 httpd  
 8064 623508 96968 httpd  
 8023 621676 97256 httpd  
 8018 623764 97312 httpd  
 7932 624012 97404 httpd  
24668 418764 97524 nova-scheduler  
 7910 624268 97704 httpd  
 7905 624268 97780 httpd  
 7998 624020 97940 httpd  
 7924 624524 97964 httpd  
 7940 624532 98088 httpd  
 7930 624524 98908 httpd  
 7936 624788 99212 httpd  
23635 501564 99348 nova-api  
23625 498908 100724 nova-api  
 8360 623484 101504 httpd  
 8382 623484 101504 httpd  
 8392 623484 101504 httpd  
 8393 623228 101504 httpd  
 8428 623484 101508 httpd  
 8336 623484 101512 httpd  
 8369 623228 101512 httpd  
 8410 623484 101512 httpd  
 8414 623484 101512 httpd  
 8429 623484 101512 httpd  
 8430 623484 101512 httpd  
 8364 623484 101576 httpd  
 8355 623484 101688 httpd  
24126 425872 101892 nova-conductor  
11851 486040 102216 neutron-server  
24120 426624 102224 nova-conductor  
24117 426184 102268 nova-conductor  
 8417 431904 102676 neutron-openvsw  
 8400 624764 102824 httpd  
 8375 624764 102832 httpd  
24134 429200 104624 nova-conductor  
24118 429096 104764 nova-conductor  
11877 460752 104912 neutron-server  
23631 492184 105708 nova-api  
11854 479508 109108 neutron-server  
23639 497816 110340 nova-api  
23624 491620 113292 nova-api  
11907 634728 114916 cinder-api  
1573 625896 116716 cinder-schedule  
11905 636640 118224 cinder-api  
11870 479048 121996 neutron-server  
11873 479144 122508 neutron-server  
11874 479148 122736 neutron-server  
11876 479384 122740 neutron-server  
11869 479176 122880 neutron-server  
11846 480812 123092 neutron-server  
11864 479536 123104 neutron-server  
11878 478772 123164 neutron-server  
11868 479592 123176 neutron-server  
11866 479500 123264 neutron-server  
11865 480768 123572 neutron-server  
11872 479972 123708 neutron-server  
11867 480160 123820 neutron-server  
11871 481428 124184 neutron-server  
23800 2086672 124280 nova-compute  
11857 481704 124496 neutron-server  
11849 486536 124628 neutron-server  
11856 490312 124884 neutron-server  
11896 652784 124936 cinder-api  
11862 481304 124972 neutron-server  
11891 639712 125112 cinder-volume  
11863 481524 125464 neutron-server  
11875 482200 126020 neutron-server  
11845 485064 126148 neutron-server  
11861 482764 126192 neutron-server  
11850 484724 126404 neutron-server  
11860 484272 126772 neutron-server  
11855 483828 126848 neutron-server  
11852 482536 127896 neutron-server  
11858 484888 128412 neutron-server  
11848 484424 128592 neutron-server  
11859 483848 128844 neutron-server  
11906 648040 130124 cinder-api  
11853 488076 131348 neutron-server  
11847 488832 133160 neutron-server  
23627 485156 134032 nova-api  
11903 653240 134540 cinder-api  
11899 651300 137632 cinder-api  
23629 498256 148388 nova-api  
4158 9024864 149912 mysqld  
23632 508104 150508 nova-api  
23634 503596 153084 nova-api  
23626 504440 153388 nova-api  
1088 1313976 157520 ovs-vswitchd  
23630 507960 157616 nova-api  
23638 512616 163256 nova-api  
23633 512444 164000 nova-api