Ask Your Question
2

Failed to allocate the network(s), not rescheduling

asked 2014-11-27 01:53:04 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Machine Setup

Host OS: ubuntu-desktop 14.04 64bit
VMware Workstation 9
OpenStack 3 node architecture all have ubuntu-server 14.04

Source: http://docs.openstack.org/juno/instal...

All seems to be working fine I don't see error on my logs but openstack fails when creating an instance. I have followed the the installation guide thoroughly, commands that I run to verify would confirm it. What could be wrong?

Command: nova service-list

+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-cert        | controller | internal | enabled | up    | 2014-11-27T07:34:46.000000 | -               |
| 2  | nova-consoleauth | controller | internal | enabled | up    | 2014-11-27T07:34:46.000000 | -               |
| 3  | nova-scheduler   | controller | internal | enabled | up    | 2014-11-27T07:34:43.000000 | -               |
| 4  | nova-conductor   | controller | internal | enabled | up    | 2014-11-27T07:34:38.000000 | -               |
| 5  | nova-compute     | compute    | nova     | enabled | up    | 2014-11-27T07:34:41.000000 | -               |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

Command: neutron agent-list

+--------------------------------------+--------------------+---------+-------+----------------+---------------------------+
| id                                   | agent_type         | host    | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+---------+-------+----------------+---------------------------+
| 418b3050-0c17-4b59-9f5e-0409e903a4f7 | Open vSwitch agent | compute | :-)   | True           | neutron-openvswitch-agent |
| 4cfe3584-931e-410e-a9eb-53ba9cef5db2 | L3 agent           | network | :-)   | True           | neutron-l3-agent          |
| 7c5617ac-08a6-475e-bfb2-70e536fbc14b | DHCP agent         | network | :-)   | True           | neutron-dhcp-agent        |
| 9a81fc03-afb7-4fe4-8e9e-0c7e9602f8f8 | Open vSwitch agent | network | :-)   | True           | neutron-openvswitch-agent |
| ad5b432b-a26e-4b31-80f1-e38bee9340bf | Metadata agent     | network | :-)   | True           | neutron-metadata-agent    |
+--------------------------------------+--------------------+---------+-------+----------------+---------------------------+

MySQL: ml2_port_bindings table content

port_id host    vif_type    driver  segment vnic_type   vif_details profile
5702fb30-770e-449c-86c3-092788d6d56b    network ovs openvswitch e09b0c77-1795-4272-8914-13e40e0b6016    normal  {"port_filter": true, "ovs_hybrid_plug": true}

LOGS:

nova-scheduler.log

2014-11-27 14:19:24.560 3062 AUDIT nova.service [-] Starting scheduler node (version 2014.2)
2014-11-27 14:19:25.542 3062 INFO oslo.messaging._drivers.impl_rabbit [req-6eabb3bd-4ad2-413a-bc7a-af16397106f6 ] Connecting to AMQP server on controller:5672
2014-11-27 14:19:25.604 3062 INFO oslo.messaging._drivers.impl_rabbit [req-6eabb3bd-4ad2-413a-bc7a-af16397106f6 ] Connected to AMQP server on controller:5672
2014-11-27 14:21:31.567 3062 INFO nova.openstack.common.periodic_task [req-e5a8c821-57c6-42ea-b7bf-680ab27417b9 None] Skipping periodic task _periodic_update_dns because its interval is negative
2014-11-27 14:21:31.651 3062 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on controller:5672
2014-11-27 14:21:31.660 3062 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on controller:5672

nova-api.log

2014-11-27 15:34:32.303 3041 INFO nova.osapi_compute.wsgi.server [req-41b7cf74-ee49-4676-b3b5-8909c5297357 None] 10.0.0.11 "GET /v2/4e01602327a342709f86d2c6ddf26107/os-services HTTP/1.1" status: 403 len: 302 time: 0.0277441
2014-11-27 15:34:47.541 3041 INFO nova.osapi_compute.wsgi.server [req-daac61a0-2b2c-4660-a42c-61f96929c314 None] 10.0.0.11 "GET /v2/60b971490ec347099907e25426f215f0/os-services HTTP/1.1" status: 200 len: 1105 time: 0.0318210

nova-compute.log

2014-11-27 15:49:03.047 6617 AUDIT nova.compute.resource_tracker [-] Total physical ram (MB): 1994, total allocated virtual ram (MB): 1024
2014-11-27 15:49:03.047 6617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 17
2014-11-27 15:49:03.048 6617 AUDIT nova.compute.resource_tracker [-] Total usable vcpus: 1, total allocated vcpus: 0
2014-11-27 15:49:03.048 6617 AUDIT nova.compute.resource_tracker [-] PCI stats: []
2014-11-27 15:49:03.048 6617 INFO nova.compute.resource_tracker [-] Compute_service record updated for compute:compute
edit retag flag offensive close merge delete

Comments

Please look at nova-cert and nova-conductor logs too. And check neutron logs on both controller and compute! Also what do you get on running 'sudo ovs-vsctl show' on all nodes? Using gre or vxlan for tenant network? I got the same error and in one of the logs i had vif_bind failed.

capsali gravatar imagecapsali ( 2014-11-27 15:31:27 -0500 )edit

My problem was that neutron didn't create the vxlan tunnel between network and compute nodes.

capsali gravatar imagecapsali ( 2014-11-27 15:32:14 -0500 )edit

http://paste.openstack.org/show/139896/ ovs-vsctl show result

I have noticed that in juno this part was skipped - icehouse install guide has it. * ovs-vsctl add-br br-int Im using gre.

Regarding vif_failed, correct me if I'm wrong mysql table ml2_binding_port should have indicated that right?

splucena gravatar imagesplucena ( 2014-11-27 18:32:40 -0500 )edit

Controller-Node logs http://paste.openstack.org/show/139917/

Compute-Node logs http://paste.openstack.org/show/139938/

Network-Node logs http://paste.openstack.org/show/139931/

splucena gravatar imagesplucena ( 2014-11-27 19:03:05 -0500 )edit

4 answers

Sort by ยป oldest newest most voted
1

answered 2015-03-04 18:57:57 -0500

muralir gravatar image

updated 2015-03-05 01:49:28 -0500

Hi - Can someone please give some hints? I was getting same error so was debugging. This is what's happening

  1. VM is spawned (libvirt call succeeds)
  2. The port get created, but throw exception vif failed waiting for call back.

2015-03-04 16:32:07.374 3053 DEBUG nova.openstack.common.loopingcall [-] Dynamic looping call <bound method="" service.periodic_tasks="" of="" <nova.service.service="" object="" at="" 0x7f56b1fc1310="">> sleeping for 52.00 seconds _inner /usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py:132 2015-03-04 16:32:38.708 3053 WARNING nova.virt.libvirt.driver [-] Timeout waiting for vif plugging callback for instance 462e1407-813c-408f-9f0a-4d2be785307c

  • The setup seems to be as per the Ubuntu Juno steps. All verified multiple times.
  • The notifications are also enabled.
  • I see the interface getting created using ovs-vsctl show after starting the nova boot command. But after 4 - 5 minutes, it throws up exception.

Update: Ok, found answer https://ask.openstack.org/en/question/26938/virtualinterfacecreateexception-virtual-interface-creation-failed/ (https://ask.openstack.org/en/question...)

Phew!!

edit flag offensive delete link more
0

answered 2019-05-13 09:01:39 -0500

Restarted Network Service and After that i am able to Launch instances on compute

edit flag offensive delete link more
0

answered 2014-11-27 08:21:11 -0500

Ram.Meena gravatar image

Hi,

Please change the debug mode 'true' in nova configuration file as below:

debug=True

Restart the nova compute service and check the logs again for more detailed error messages. You may also provide the logs at http://paste.openstack.org/ .

You will need to set the 'virt_type = qemu' if you are using vmware. Make sure that your installation is able to allocate appropriate resources to newly launched instances, you may try to launch the instances using 'cirros' image which needs very less amount of resources.

edit flag offensive delete link more

Comments

virt_type = qemu? Even if egrep -c '(vms|svm)' /proc/cpuinfo returned 1?

splucena gravatar imagesplucena ( 2014-11-27 18:34:28 -0500 )edit
0

answered 2014-12-05 01:23:49 -0500

Huaming gravatar image

I've encountered the same issue, no idea.

root@compute1:~# more /var/log/nova/nova-compute.log |grep ERR 2014-12-05 06:49:42.645 9310 ERROR nova.compute.manager [req-75358cd3-5fda-488d-b818-ebe0e07a276e None] [instance: 467e58d6-f8e0-4c01-9263-810c4b8286c9] Instance failed to spawn

2014-12-05 06:49:43.000 9310 ERROR nova.compute.manager [req-75358cd3-5fda-488d-b818-ebe0e07a276e None] [instance: 467e58d6-f8e0-4c01-9263-810c4b8286c9] Failed to allocate network(s)

2014-12-05 06:49:43.004 9310 ERROR nova.compute.manager [req-75358cd3-5fda-488d-b818-ebe0e07a276e None] [instance: 467e58d6-f8e0-4c01-9263-810c4b8286c9] Build of instance 467e58d6-f8e0-4c01-9263-810c4b8286c9 aborted: Failed to allocate the network(s), not rescheduling. root@compute1:~#

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

Stats

Asked: 2014-11-27 01:53:04 -0500

Seen: 19,616 times

Last updated: May 13 '19