No more free ram in my controller node

asked 2019-01-25 06:35:58 -0600

najeh gravatar image

updated 2019-01-25 10:30:56 -0600

After a month of openstack installation and creation and deletion of instances. Now i couldn't create any more, and i got this error "{u'message': u'No valid host was found. ', u'code': 500, u'created': u'2019-01-22T15:14:31Z'} "

Ps: i have one controller node and two compute nodes.

By executing the below command on my controller node i got "free_ram_mb" is equal to -1510

root@controller:~# nova hypervisor-stats

+----------------------+-------+

| Property | Value |

+----------------------+-------+

| count | 2 |

| current_workload | 0 |

| disk_available_least | 860 |

| free_disk_gb | 752 |

| free_ram_mb | -1510 |

| local_gb | 912 |

| local_gb_used | 6 |

| memory_mb | 15898 |

| memory_mb_used | 1026 |

| running_vms | 7 |

| vcpus | 12 |

| vcpus_used | 0 |

+----------------------+-------+

root@controller:~# free -mh

           total                        utilisé      libre           partagé    tamp/cache   disponible Mem:   7.8G                       4.6G           553M         54M           2.6G                2.8G Partition d'échange:        975M        9.1M        966M




#root@compute1:~# free -mh
          total       utilisé      libre     partagé tamp/cache   disponible Mem:           7.8G        444M        6.7G        1.0M       610M        7.1G Partition d'échange:        975M          0B        975M



#root@compute2:~# free -mh
          total       utilisé      libre     partagé tamp/cache   disponible Mem:           7.8G        446M        6.7G        1.0M        631M        7.1G Partition d'échange:        975M          0B        975M

root@controller:~# openstack host show compute1

+----------+------------+-----+-----------+---------+

| Host | Project | CPU | Memory MB | Disk GB |

+----------+------------+-----+-----------+---------+

| compute1 | (total) | 6 | 7949 | 456 |

| compute1 | (used_now) | 0 | 512 | 3 |

| compute1 | (used_max) | 0 | 0 | 0 |

+----------+------------+-----+-----------+---------+

root@controller:~# openstack host show compute2

+----------+------------+-----+-----------+---------+ | Host |Project | CPU | Memory MB | Disk GB | +----------+------------+-----+-----------+---------+ | compute2 | (total) | 6 | 7949 | 456 | | compute2 | (used_now) | 0 | 514 | 3 | | compute2 | (used_max) | 0 | 0 | 0 |+----------+------------+-----+-----------+---------+

root@controller:~# openstack host show controller

many found

root@controller:~# openstack usage list

Usage from 2018-12-28 to 2019-01-26:

+-----------+---------+--------------+-----------+---------------+

| Project | Servers | RAM MB-Hours | CPU Hours | Disk GB-Hours |

+-----------+---------+--------------+-----------+---------------+

| myproject | 46 | 5084334.44 | 2486.24 | 49653.12 |

| admin | 14 | 100582.4 | 49.11 | 982.25 |

+-----------+---------+--------------+-----------+---------------+

virt-host-validate

QEMU: Checking for device assignment IOMMU support : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)

How can i solve the problem and get more free ram?

ps: all created instances are deleted

edit retag flag offensive close merge delete

Comments

1

Does your controller also serve as compute node? Because the nova hypervisor-stats command shows the usage of your compute nodes. So your compute nodes don't have enough ram. If all instances are deleted the ram should usually be available, I would check with top what processes consume the ram.

eblock gravatar imageeblock ( 2019-01-25 07:16:48 -0600 )edit

@eblock I have one controller node and two compute nodes.

I deleted all instances.

how can i check with top what processes consume the ram ?

I added above in the question description some informations about compute and controller nodes

najeh gravatar imagenajeh ( 2019-01-25 07:40:46 -0600 )edit

@Bernd Bausch any suggestion please?

najeh gravatar imagenajeh ( 2019-01-26 01:18:20 -0600 )edit
1

As always: Use the scheduler log to find out if the Placement service or a filter causes the scheduling failure. Use debug logging. If it’s the Placement service, the Placement CLI tells you what’s in the Placement DB.

Bernd Bausch gravatar imageBernd Bausch ( 2019-01-26 02:36:18 -0600 )edit

@Bernd Bausch In /var/log/nova/nova-scheduler.log file i found this error:

Failed to retrieve allocation candidates from placement API for filters: RequestGroup(use_same_provider=False, resources={DISK_GB:1, MEMORY_MB:64, VCPU:1}, traits=[], aggregates=[])

najeh gravatar imagenajeh ( 2019-01-26 03:48:09 -0600 )edit