Ask Your Question
3

error in creating instances "no valid host was found"

asked 2014-11-21 06:39:04 -0500

scream gravatar image

updated 2014-12-31 14:21:46 -0500

smaffulli gravatar image

I used devstack to deploy a all-in-one openstack in one VMware ubuntu14.04 virtual machine, which is configured with VT-x/EPT enabled so as to inherit the virtualization features of the host machine, which is a windows 7 x64 edition.

The configuration of localrc is as follows, with Ironic and Ceilometer enabled.

# Credentials
ADMIN_PASSWORD=password
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password

# Enable Ironic API and Ironic Conductor
enable_service ironic
enable_service ir-api
enable_service ir-cond

# Enable Neutron which is required by Ironic and disable nova-network.
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron

# Create 3 virtual machines to pose as Ironic's baremetal nodes.
IRONIC_VM_COUNT=3
IRONIC_VM_SSH_PORT=22
IRONIC_BAREMETAL_BASIC_OPS=True

# The parameters below represent the minimum possible values to create
# functional nodes.
IRONIC_VM_SPECS_RAM=1024
IRONIC_VM_SPECS_DISK=10

# Size of the ephemeral partition in GB. Use 0 for no ephemeral partition.
IRONIC_VM_EPHEMERAL_DISK=0

VIRT_DRIVER=ironic

# By default, DevStack creates a 10.0.0.0/24 network for instances.
# If this overlaps with the hosts network, you may adjust with the
# following.
NETWORK_GATEWAY=10.1.0.1
FIXED_RANGE=10.1.0.0/24
FIXED_NETWORK_SIZE=256

# Log all output to files
LOGFILE=$HOME/devstack.log
SCREEN_LOGDIR=$HOME/logs
IRONIC_VM_LOG_DIR=$HOME/ironic-bm-logs

# Enable the ceilometer metering services
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector

# Enable the ceilometer alarming services
enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier

# Enable the ceilometer api services
enable_service ceilometer-api

So I run /tools/create-stack-user.sh and changed to stack user, then run ./stack.sh to start the long journey to a final success in everything installed as follows:

> Horizon is now available at
> http://192.168.209.252/ Keystone is
> serving at
> http://192.168.209.252:5000/v2.0/
> Examples on using novaclient command
> line is in exercise.sh The default
> users are: admin and demo The
> password: password This is your host
> ip: 192.168.209.252

It's very nice to login in the dashboard and see everything shown up. However, when I tried to create an instance, it always pops up a warning after some time of initializaing.

[error: No valid host was found. Exceeded max scheduling attempts 3 for instance 2203febf-2ef0-4de5-a176-71648bf87b09. Last exception: [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2028, in _do_build_and_run_in].

or sometimes another error:

No valid host was found. There are not enough hosts available.
500
File "/opt/stack/nova/nova/conductor/manager.py", line 626, in build_instances request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 49, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/client/__init__.py", line 35, in __run_method return getattr(self.instance, __name)(*args, **kwargs) File "/opt/stack/nova/nova/scheduler/client/query.py", line 34, in select_destinations context, request_spec, filter_properties) File "/opt/stack/nova/nova/scheduler/rpcapi.py", line 112, in select_destinations request_spec=request_spec, filter_properties=filter_properties) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call retry=self.retry) File "/usr/local/lib/python2.7 ...
(more)
edit retag flag offensive close merge delete

Comments

1

can you paste the nova-scheduler.log and nova-compute.log to paste.openstack.org? these two logs are important to resolve the issue.

9lives gravatar image9lives ( 2014-11-22 00:31:44 -0500 )edit

Actually there is no seperate log files such as nova-scheduler.log and nova-compute.log with devstack running everything in screen. So I just paste all screen logs in /opt/stack on http://pan.baidu.com/s/1dDEhC4L . You can find something useful there.

scream gravatar imagescream ( 2014-11-23 18:47:04 -0500 )edit

The name of the needed log file is screen-n-sch.log . It is available on your file host but with a terrible download rate. Could you paste it e.g. to http://paste.openstack.org/about/ ? If the content of the file is too long, use grep ERROR screen-n-sch.log -A 100 -B 50.

Markus Zoeller gravatar imageMarkus Zoeller ( 2014-11-25 03:18:27 -0500 )edit
scream gravatar imagescream ( 2014-11-25 18:38:52 -0500 )edit

could you also do further searching by grep -iE 'error|warning' screen-n-sch.log -C 5 -5 and paste to the paste.openstack.org , i guess there must be some warning on nova-scheduler log.

9lives gravatar image9lives ( 2014-11-25 21:21:33 -0500 )edit

6 answers

Sort by » oldest newest most voted
1

answered 2014-11-26 10:26:08 -0500

SysFiller gravatar image

If you are running everything on a virtual machine, did you set virt_type=qemu in /etc/nova/nova.conf ( [libvirt] section )?

edit flag offensive delete link more

Comments

Yes, I did it

scream gravatar imagescream ( 2014-11-27 02:37:47 -0500 )edit

are you sure you don't mean /etc/nova/nova-compute.conf - on the compute node.

MarkChu gravatar imageMarkChu ( 2015-07-20 13:22:21 -0500 )edit
1

answered 2014-11-26 04:40:10 -0500

updated 2014-11-26 07:53:07 -0500

A few observations of your log files:

Observation 1 : Database consistency

It seems that there could be some inconsistence between database records and actual environment:

grep -E 'WARNING\b' screen-n-cpu.log -C 5 -5

shows:

2014-11-23 16:23:53.648 WARNING nova.compute.manager [-] While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor.

Possible approach: A retry of ./unstack.sh and ./stack.sh could solve that. AFAIK the database will be reset during this.

Observation 2 : Scheduler Filtering

It seems that the scheduler starts with 3 hosts. Two of them will be filtered by the "retry filter":

grep -E 'nova.filters\b' screen-n-sch.log -C 5 -5

shows:

2014-11-21 04:22:10.081 DEBUG nova.filters [req-4df8086c-dd0f-4c2b-86ca-47d737629a05 admin admin] Starting with 3 host(s) from (pid=15141) get_filtered_objects /opt/stack/nova/nova/filters.py:70

2014-11-21 04:22:10.088 DEBUG nova.scheduler.filters.retry_filter [req-4df8086c-dd0f-4c2b-86ca-47d737629a05 admin admin] Host [u'ubuntu', u'8f97a0c0-1766-4fd1-aea4-a962a250e59e'] fails.  Previously tried hosts: [[u'ubuntu', u'8f97a0c0-1766-4fd1-aea4-a962a250e59e'], [u'ubuntu', u'339588f0-56bd-49bd-8974-90c7abfde60e']] from (pid=15141) host_passes /opt/stack/nova/nova/scheduler/filters/retry_filter.py:42
2014-11-21 04:22:10.089 DEBUG nova.scheduler.filters.retry_filter [req-4df8086c-dd0f-4c2b-86ca-47d737629a05 admin admin] Host [u'ubuntu', u'339588f0-56bd-49bd-8974-90c7abfde60e'] fails.  Previously tried hosts: [[u'ubuntu', u'8f97a0c0-1766-4fd1-aea4-a962a250e59e'], [u'ubuntu', u'339588f0-56bd-49bd-8974-90c7abfde60e']] from (pid=15141) host_passes /opt/stack/nova/nova/scheduler/filters/retry_filter.py:42

More information about novas filters can be found at [1].

One host passes this filtering process (id 057b5bf9-faa1-4e62-a8e0-9f38071dae3d) and should be a valid target for the placement of the launched instance. But it seems it isn't used, which confuses me.

Possible approach: Add the AllHostsFilter to the configuration parameter scheduler_default_filters in the /etc/nova/nova.conf file. This assumes that the configuration parameter scheduler_use_baremetal_filters is still on its default False. For more information about these parameters see Table 2.51. Description of scheduler configuration options in [2].

Observation 3 : Resources

The hosts seem to have enough resources for the instance, but maybe more memory and CPUs would be a good idea. The host I reference is:

screen-n-sch.2014-11-21-031146.log:2014-11-21 04:22:04.911 DEBUG nova.scheduler.filter_scheduler [req-4df8086c-dd0f-4c2b-86ca-47d737629a05 admin admin] Weighed [WeighedHost [host: (ubuntu, 339588f0-56bd-49bd-8974-90c7abfde60e) ram:1024 disk:10240 io_ops:0 instances:0, weight: 1.0], WeighedHost [host: (ubuntu, 057b5bf9-faa1-4e62-a8e0-9f38071dae3d) ram:1024 disk:10240 io_ops:0 instances:0, weight: 1.0]] from (pid=15141) _schedule /opt/stack/nova/nova/scheduler/filter_scheduler.py:263

Conclusion

All I could provide with my current knowledge are these observations and possible approaches, sorry.

References

edit flag offensive delete link more

Comments

Thanks very much for your detailed analysis. The first problem is solved by retrying of ./unstack.sh and ./stack.sh. I added the AllHostsFilter to the configuration parameter scheduler_default_filters in the /etc/nova/nova.conf file as you said and there is no such warning in the new generated logs.

scream gravatar imagescream ( 2014-11-26 07:53:27 -0500 )edit

The thrid one shouldn't be a problem as I adjust the memory and disk to larger ones, the problem "Exceeded max scheduling attempts" still remains. Maybe there are other unknown problems. Thank you anyway :)

scream gravatar imagescream ( 2014-11-26 07:56:26 -0500 )edit

Maybe it's because I installed devstack on virtual machines?

scream gravatar imagescream ( 2014-11-26 07:57:16 -0500 )edit

You want to use the baremetal service ironic, one could think that this could be a problem in VMs. But if this would be the case, the OpenStack functional tests wouldn't be possible then. In other words, I don't think this is a "devstack in VM" issue.

Markus Zoeller gravatar imageMarkus Zoeller ( 2014-11-26 08:02:56 -0500 )edit
0

answered 2014-11-21 23:23:00 -0500

yatin gravatar image

It seems nova-compute is not started, check nova-compute is started using: service nova-compute status, also check what nova-processes are running using ps -aef | grep nova Also run the following command to check nova services list nova service-list If this not solves check nova-compute logs: nova-compute.log and share nova.conf and nova-compute.conf files.

edit flag offensive delete link more

Comments

The checking status is pasted in http://paste.openstack.org/show/137092/ . It seems like nova-compute is enabled and working. And detailed logs are pasted in http://pan.baidu.com/s/1dDEhC4L . Thanks

scream gravatar imagescream ( 2014-11-23 18:53:40 -0500 )edit
0

answered 2014-11-25 23:08:29 -0500

Ram.Meena gravatar image

Hi,

Since there is a possibility that hypervisor on underlying vmware VM is not working appropriately so I would suggest you to check if you can create VMs using 'virt-install' command and check if your hypervisor (KVM/QEMU) is working or not. 'virt-install' is a command line tool for provisioning new virtual machines using the "libvirt" hypervisor management library. You may also use the 'virt-manager' if you need GUI to create VM. 'virt-manager' can be used to manage the created virtual machines via GUI and to access the consoles.

edit flag offensive delete link more

Comments

Actually I can create virtual machines use kvm which called functions of libvirt, so I believe it's not the problem.

scream gravatar imagescream ( 2014-11-26 05:58:41 -0500 )edit
0

answered 2015-04-21 13:40:43 -0500

badmadrad gravatar image

My issue was because I did not have enough ram allocated to nova-scheduler on my bare-metal server. I wasn't afraid of overcommitment since it was dev machine.

Open:

sudo vim /etc/nova/nova.conf

Change: ram_allocation_ratio=1.5 to ram_allocation_ratio=3

Restart:

sudo service openstack-nova-scheduler restart
sudo service openstack-nova-conductor restart
edit flag offensive delete link more
0

answered 2015-05-21 01:23:20 -0500

检查logs目录下的出错信息,我的原因是BIOS没有开启VT,kvm模块未加载,请按照以下步骤查看是否配置正确。 https://wiki.ubuntu.com/kvm http://docs.openstack.org/juno/config-reference/content/kvm.html (http://docs.openstack.org/juno/config...)

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

6 followers

Stats

Asked: 2014-11-21 06:39:04 -0500

Seen: 24,453 times

Last updated: Dec 31 '14