Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

VM creation fail : No vaild host was found

Hi. I'm deploying Openstack Mitaka with 1 controller and 1networknode and 3compute node. Everything seems to work fine, and I'm able to launch instances also. But now I'm trying to launch instance with large capacity options, like m1.xlarge, and it fails, saying No valid host was found. I found out that Filters filtering out all my compute nodes as a valid host. And I believe overcommit isn't really a way to get this done, since it might crash when disk or ram of that node are fully used. So my questions..

  1. Do I have to assign RAM or diskstorage constrained to the amount of the maximum of one compute node? If so, please tell me the reason? I'm confused.. isn't that against the concept of cloud?
  2. my hypervisor-stats shows all my three compute-nodes, but filters are not counting compute-node2 as you can see in the error log. What configurations or files should I be looking at?

tail /var/log/nova/nova-scheduler.log - diskfilter

 2016-11-10 18:53:12.288 10845 DEBUG nova.scheduler.host_manager [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Update host state with aggregates: [] _locked_update /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:171
    2016-11-10 18:53:12.288 10845 DEBUG nova.scheduler.host_manager [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Update host state with service dict: {'binary': u'nova-compute', 'deleted': False, 'created_at': datetime.datetime(2016, 10, 15, 7, 31, 22, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2016, 11, 10, 9, 53, 11, tzinfo=<iso8601.Utc>), 'report_count': 12760, 'topic': u'compute', 'host': u'compute-node3', 'version': 9, 'disabled': False, 'forced_down': False, 'last_seen_up': datetime.datetime(2016, 11, 10, 9, 53, 11, tzinfo=<iso8601.Utc>), 'deleted_at': None, 'disabled_reason': None, 'id': 8} _locked_update /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:174
    2016-11-10 18:53:12.289 10845 DEBUG nova.scheduler.host_manager [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Update host state with instances: {} _locked_update /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:177
    2016-11-10 18:53:12.289 10845 DEBUG oslo_concurrency.lockutils [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Lock "(u'compute-node3', u'compute-node3')" released by "nova.scheduler.host_manager._locked_update" :: held 0.003s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
    2016-11-10 18:53:12.290 10845 WARNING nova.scheduler.host_manager [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] No compute service record found for host compute-node2
    2016-11-10 18:53:12.290 10845 DEBUG nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70
    2016-11-10 18:53:12.291 10845 DEBUG nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter RetryFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
    2016-11-10 18:53:12.291 10845 DEBUG nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
    2016-11-10 18:53:12.292 10845 DEBUG nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter RamFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
    2016-11-10 18:53:12.292 10845 DEBUG nova.scheduler.filters.disk_filter [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] (compute-node, compute-node) **ram: 7912MB disk: 65536MB io_ops: 0 instances: 2 does not have 102400 MB usable disk,** it only has 65536.0 MB usable disk. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/disk_filter.py:55
    2016-11-10 18:53:12.292 10845 DEBUG nova.scheduler.filters.disk_filter [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] (compute-node3, compute-node3) r**am: 9386MB disk: 29696MB io_ops: 0 instances: 0 does not have 102400 MB usable disk,** it only has 29696.0 MB usable disk. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/disk_filter.py:55
    2016-11-10 18:53:12.293 10845 INFO nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter DiskFilter returned 0 hosts
    2016-11-10 18:53:12.293 10845 DEBUG nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filtering removed all hosts for the request with instance ID '3018dd57-a9da-4451-95e3-441b4e690bb6'. Filter results: [('RetryFilter', [(u'compute-node', u'compute-node'), (u'compute-node3', u'compute-node3')]), ('AvailabilityZoneFilter', [(u'compute-node', u'compute-node'), (u'compute-node3', u'compute-node3')]), ('RamFilter', [(u'compute-node', u'compute-node'), (u'compute-node3', u'compute-node3')]), ('DiskFilter', None)] get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:129
    2016-11-10 18:53:12.294 10845 INFO nova.filters [req-10c398e2-6611-4884-8883-cdd4c2445848 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filtering removed all hosts for the request with instance ID '3018dd57-a9da-4451-95e3-441b4e690bb6'. Filter results: ['RetryFilter: (start: 2, end: 2)', 'AvailabilityZoneFilter: (start: 2, end: 2)', 'RamFilter: (start: 2, end: 2)'**, 'DiskFilter: (start: 2, end: 0)']**

another scheduler log - ramfilter

2016-11-11 13:14:16.038 1214 DEBUG nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Starting with 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70
2016-11-11 13:14:16.039 1214 DEBUG nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter RetryFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2016-11-11 13:14:16.039 1214 DEBUG nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter AvailabilityZoneFilter returned 2 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104
2016-11-11 13:14:16.040 1214 DEBUG nova.scheduler.filters.ram_filter [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] (compute-node, compute-node) ram: 7912MB disk: 167936MB io_ops: 0 instances: 2 does not have 14336 MB usable ram before overcommit, it only has 9960 MB. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/ram_filter.py:45
2016-11-11 13:14:16.040 1214 DEBUG nova.scheduler.filters.ram_filter [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] (compute-node3, compute-node3) ram: 9386MB disk: 152576MB io_ops: 0 instances: 0 does not have 14336 MB usable ram before overcommit, it only has 9898 MB. host_passes /usr/lib/python2.7/dist-packages/nova/scheduler/filters/ram_filter.py:45
2016-11-11 13:14:16.041 1214 INFO nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filter RamFilter returned 0 hosts
2016-11-11 13:14:16.041 1214 DEBUG nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filtering removed all hosts for the request with instance ID '84711475-309f-4f73-b541-4389a60833dd'. Filter results: [('RetryFilter', [(u'compute-node', u'compute-node'), (u'compute-node3', u'compute-node3')]), ('AvailabilityZoneFilter', [(u'compute-node', u'compute-node'), (u'compute-node3', u'compute-node3')]), ('RamFilter', None)] get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:129
2016-11-11 13:14:16.041 1214 INFO nova.filters [req-3721709a-224f-4907-9205-b3bf75d78e7a 6abfef77e9984fc58bb5decc566d12c2 fda14487a68f436da7b55fe218798ea2 - - -] Filtering removed all hosts for the request with instance ID '84711475-309f-4f73-b541-4389a60833dd'. Filter results: ['RetryFilter: (start: 2, end: 2)', 'AvailabilityZoneFilter: (start: 2, end: 2)', 'RamFilter: (start: 2, end: 0)']

nova hypervisor-stats

+----------------------+-------+
| Property             | Value |
+----------------------+-------+
| count                | 3     |
| current_workload     | 0     |
| disk_available_least | 292   |
| free_disk_gb         | 418   |
| free_ram_mb          | 26746 |
| local_gb             | 447   |
| local_gb_used        | 29    |
| memory_mb            | 29818 |
| memory_mb_used       | 3072  |
| running_vms          | 2     |
| vcpus                | 12    |
| vcpus_used           | 3     |
+----------------------+-------+