Openstack-nova-scheduler service cannot any longer connect to AMQP Server performing "nova boot .." on Fedora 20

asked 2014-02-11 05:31:30 -0500

dbaxps

10778 ●57 ●112 ●224 http://bderzhavets.blo...

updated 2014-02-11 13:05:26 -0500

I have 2 Two Physical Node F20 Neutron GRE+OVS clusters been set up per instructions :-

First instruction

Second instruction

First one was running fine since 01/23/14. Second one just for 5-6 hours through this night.
Now when I attempt to  run something like ( what worked for 2 weeks ) :-

[root@dfw02 (keystone_admin)]$ nova boot --flavor 2  --user-data=./myfile.txt --block_device_mapping vda=4cb4c501-c7b1-4c42-ba26-0141fcde038b:::0 VF20

I get in /var/log/nova/scheduler.log

2014-02-11 13:29:37.718 1161 ERROR nova.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: connection aborted. Sleeping 1 seconds
2014-02-11 13:32:12.643 1161 WARNING nova.scheduler.driver [req-227e92ad-38f4-44a3-b986-8754c017e9b9 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 934b661f-3d18-4a3e-9fdc-ca3458be61cb] Setting instance to ERROR state.
2014-02-11 13:38:58.537 1161 WARNING nova.scheduler.driver [req-7a081f55-9e8d-41a1-9ce2-0a16c46831d6 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: bc63d821-f203-4344-8a90-f9c9760068ee] Setting instance to ERROR state.
2014-02-11 04:58:13.619 1161 WARNING nova.scheduler.driver [req-914688db-33e5-4c8c-8824-99caab91e7ba 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 34443dab-f1f9-4a00-bfb6-636b85bbbef4] Setting instance to ERROR state.
2014-02-11 05:32:45.815 1161 WARNING nova.scheduler.driver [req-1cbbd878-6358-4de5-a805-8017f15a4024 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: bd409f62-e7ce-4fb9-92a9-1a558c5fe396] Setting instance to ERROR state.
2014-02-11 14:35:39.835 1161 WARNING nova.scheduler.driver [req-ca8dc850-1385-4128-8f93-614a7c01641e 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 90f7c2cb-8044-4cc1-8f9f-a7e85ce35a51] Setting instance to ERROR state.
2014-02-11 14:37:41.206 1161 WARNING nova.scheduler.driver [req-d67abe4d-3b15-4aa3-a9d7-13f302c7d039 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 90ac3c0d-643c-479f-a1bc-6a126efc9a25] Setting instance to ERROR state.

All suspended instances of Ubuntu 13.10 and Fedora 20 may be resumed. But I am unable to create any new instance ( even Cirros) it immediately gets ERROR state

Second Cluster was done yesterday night and was alive just for several hours. First one was down. But this morning
it started behave the same way as second at about 2 a.m. ,  even  before 'yum -y update'.

Same log on another controller :-

2014-02-11 03:06:48.050 767 ERROR nova.openstack.common.rpc.impl_qpid [req-05e0de62-9a83-47fe-b24d-1fbfd0d2b0a0 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
2014-02-11 03:07:57.333 747 WARNING nova.openstack.common.db.sqlalchemy.session [req-556615f6-9274-45da-8135-9e40e73e5642 None None] SQL connection failed. infinite attempts left.
2014-02-11 03:09:47.979 747 WARNING nova.scheduler.driver [req-6f2a0473-7974-4cdb-8695-d9d170fce653 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 2dbebc5f-ead6-4aa1-93b1-bf66c5809959] Setting instance to ERROR state.
2014-02-11 03:16:20.056 747 WARNING nova.scheduler.driver [req-fee47a38-b5dc-450c-8fc5-865fd6155910 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: fa298251-10fe-43e1-b709-38bd47393bb4] Setting instance to ERROR state.
2014-02-11 03:18:19.799 745 WARNING nova.openstack.common.db.sqlalchemy.session [req-f1d2d706-0b26-401e-9e31-c4e23b583da4 None None] SQL connection failed. infinite attempts left.
2014-02-11 03:20:07.948 745 WARNING nova.scheduler.driver [req-a5d21042-0a31-47a5-a1d6-aff761bd4598 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 06e477e6-6eae-43b2-95d3-a4473f7add2e] Setting instance to ERROR state.
2014-02-11 11:42:32.059 762 WARNING nova.openstack.common.db.sqlalchemy.session [req-69ee79ff-3ffc-4acc-9d0e-67b99f061610 None None] SQL connection failed. infinite attempts left.
2014-02-11 12:47:17.438 762 WARNING nova.scheduler.driver [req-ba07c02f-cf9a-462d-a337-8df13e0c2926 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: db57afc6-e980-45fc-a8ba-fcb64699d4a4] Setting instance to ERROR state.
2014-02-11 12:55:20.508 762 WARNING nova.scheduler.driver [req-cc5b29e5-8819-4912-94db-46e9558ae65b 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 6c3ec297-2bcd-4ea6-954a-d7aa0f20dbe6] Setting instance to ERROR state.
2014-02-11 17:47:17.533 751 WARNING nova.openstack.common.db.sqlalchemy.session [req-2cefe83c-4117-4212-825a-bf095617769f None None] SQL connection ...

(more)

edit retag flag offensive close merge delete

Comments

View also https://bugzilla.redhat.com/show_bug....

dbaxps ( 2014-05-28 01:50:18 -0500 )edit

add a comment

answered 2014-02-17 07:02:13 -0500

dbaxps

10778 ●57 ●112 ●224 http://bderzhavets.blo...

updated 2014-02-22 07:21:11 -0500

Looks like  RDO Openstack Nova Software evaluated that my Compute Node may run at a time no more  then 5 cloud instances. So , to create new one I just must have in `nova list` <= then 4 entries. Then  I will be able create new one instance for sure. No new errors appear in scheduler.log.  It has been tested on 2 "Two Node Neutron GRE+OVS+GlusterBackend for Cinder  Systems"  ( watching in other window `tail -f /var/log/nova/scheduler.log` for ERRORS to connect AMQP Server). However, it is still  a guess based on more then  10 new ( created after limit overcome and removing one existing vm )  instances loaded via `nova boot` (tested cinder volumes on glusterfs (ext4fs) and glance images).  Log /var/log/nova/scheduler.log is actually misleading. It reacts on `nova boot` failure as AMQP connection error.

Another option, it's related with nova quota-show for tenant. (10 default). Having 3 vms on Compute I brought up openstack-nova-compute on Controller and was able to create 2 vms more on Compute and 5 vms on Controller.

However, before adding vms I ran

 # nova-manage project quota --project=abb1fa95b0ec448ea8da3cc99d61d301  --key=instances --value=20
    I got 
    [root@dallas1 ~(keystone_admin)]$ mysql -u root -p
    Enter password: 
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MariaDB connection id is 17
    Server version: 5.5.34-MariaDB MariaDB Server

    Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.

    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

    MariaDB [(none)]> use nova;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A

    Database changed
    MariaDB [nova]> SELECT * FROM quotas;
    +----+---------------------+------------+------------+----------------------------------+-----------+------------+---------+
    | id | created_at          | updated_at | deleted_at | project_id                         | resource  | hard_limit | deleted |
    +----+---------------------+------------+------------+----------------------------------+-----------+------------+---------+
    |  1 | 2014-02-22 11:33:40 | NULL | NULL | abb1fa95b0ec448ea8da3cc99d61d301 | instances | 20 |       0 |
    +----+---------------------+------------+------------+----------------------------------+-----------+------------+---------+
    1 row in set (0.00 sec)

Actually , it didn't  change  instances limit and `nova quota-show` output as well.
In meantime I am pretty sure that my actions just attempt to overcome some quota. By some reason  scheduler.log reports the same
2014-02-22 15:12:34.008 1133 ERROR nova.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: connection aborted. Sleeping 1 seconds
2014-02-22 15:41:53.345 1133 WARNING nova.scheduler.driver [req-5d560132-9399-40b5-9479-047350ff761b abb1fa95b0ec448ea8da3cc99d61d301 ab1cd5ee334a4caeafdb2df90540359a] [instance: 54778c9f-96bb-46d2-b0f1-af2c6d19e02a] Setting instance to ERROR state.
2014-02-22 16:04:31.064 1133 WARNING nova.scheduler.driver [req-9fd21222-51fa-4873-a914-5addb29c125e abb1fa95b0ec448ea8da3cc99d61d301 ab1cd5ee334a4caeafdb2df90540359a] [instance: 11fcd7e4-9141-4bb4-9e2f-b27ad8ff0f07] Setting instance to ERROR state.

edit flag offensive delete link

add a comment

Openstack-nova-scheduler service cannot any longer connect to AMQP Server performing "nova boot .." on Fedora 20

Comments

1 answer

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Openstack-nova-scheduler service cannot any longer connect to AMQP Server performing "nova boot .." on Fedora 20 edit

Comments

1 answer

Your Answer

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Openstack-nova-scheduler service cannot any longer connect to AMQP Server performing "nova boot .." on Fedora 20