Can't spawn instances due to no ram

asked 2018-07-24 08:08:10 -0500

hudac gravatar image

I was able to install openstack on centos using packstack (this https://www.tecmint.com/openstack-installation-guide-rhel-centos/ (manual)) (I have one machine that run all openstack modules - hypervisor and compute).

I was able to spawn ubuntu16.04 cloud image once, with flavor m1.small.
I had to delete it and create a new instance because of wrong key pair.
Since then, I couldn't spawn any instance.

I was able to see in nova-compute.log the next error message:

qemu-kvm: cannot set up guest memory 'pc.ram': Cannot allocate memory

free -h

          total        used        free      shared  buff/cache   available
Mem:            15G         14G        222M         54M        356M        120M
Swap:          6.8G        6.8G         20K

I don't understand where all the RAM went.

This is what top show (no one use that much RAM).

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                 
733 apache    20   0 1495388 253644   2876 S   5.6  1.6   3:02.59 httpd                                                                                                   
2129 mysql     20   0 9079668 250452   3616 S  27.8  1.5 211:05.47 mysqld                                                                                                  
734 apache    20   0 1371060 171732   2328 S   0.0  1.1   1:02.65 httpd                                                                                                   
23638 nova      20   0  513640 163888   1632 S   0.0  1.0   0:21.08 nova-api                                                                                                
23633 nova      20   0  512032 163560   2012 R  88.9  1.0  44:30.40 nova-api                                                  
...

I don't think that's the case of linux use the memory for io cache (as described https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/chap-oracle_9i_and_10g_tuning_guide-memory_usage_and_page_cache (here)) because you can see the buff/cache is only 356M.

On the other hand, this is what openstack hypervisor show 1 -f json show:

...
  "memory_mb_used": 16016, 
  "uptime": "5 days,  3:05", 
  "host_time": "08:02:38", 
  "local_gb_used": 12, 
  "id": 1, 
  "current_workload": 0, 
  "state": "up", 
  "load_average": "5.01, 9.39, 6.09", 
  "users": "4", 
  "aggregates": [], 
  "host_ip": "192.168.201.209", 
  "hypervisor_hostname": "cloud.centos.lan", 
  "hypervisor_version": 2010000, 
  "disk_available_least": 27, 
  "local_gb": 40, 
  "free_ram_mb": 15861, 
  "vcpus_used": 0, 
  "hypervisor_type": "QEMU", 
  "memory_mb": 16373, 
  "vcpus": 16, 
  "running_vms": 0, 
  "service_id": 9, 
  "service_host": "cloud.centos.lan"
  • Is there something I can do in order to start instances?
  • What is taking all this RAM?
  • I tried flushing https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/ (Linux io cache) without any help

Why there are so many processes that take all of my physical RAM ?

 ps -e -o pid,vsz,rss,comm= | sort -n -k 3  
 ...  
 7999 623252 96480 httpd  
 8019 623252 96648 httpd  
 8068 623252 96824 httpd  
 7920 623500 96960 httpd  
 8064 623508 96968 httpd  
 8023 621676 97256 httpd  
 8018 623764 97312 httpd  
 7932 624012 97404 httpd  
24668 418764 97524 nova-scheduler  
 7910 624268 97704 httpd  
 7905 624268 97780 httpd  
 7998 624020 97940 httpd  
 7924 624524 97964 httpd  
 7940 624532 98088 httpd  
 7930 624524 98908 httpd  
 7936 624788 99212 httpd  
23635 501564 99348 nova-api  
23625 498908 100724 nova-api  
 8360 623484 101504 httpd  
 8382 623484 101504 httpd  
 8392 623484 101504 httpd  
 8393 623228 101504 httpd  
 8428 623484 101508 httpd  
 8336 623484 101512 httpd  
 8369 623228 101512 httpd  
 8410 623484 101512 httpd  
 8414 623484 ...
(more)
edit retag flag offensive close merge delete