Openstack-nova-scheduler service cannot any longer connect to AMQP Server performing "nova boot .." on Fedora 20
I have 2 Two Physical Node F20 Neutron GRE+OVS clusters been set up per instructions :-
First one was running fine since 01/23/14. Second one just for 5-6 hours through this night.
Now when I attempt to run something like ( what worked for 2 weeks ) :-
[root@dfw02 (keystone_admin)]$ nova boot --flavor 2 --user-data=./myfile.txt --block_device_mapping vda=4cb4c501-c7b1-4c42-ba26-0141fcde038b:::0 VF20
I get in /var/log/nova/scheduler.log
2014-02-11 13:29:37.718 1161 ERROR nova.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: connection aborted. Sleeping 1 seconds
2014-02-11 13:32:12.643 1161 WARNING nova.scheduler.driver [req-227e92ad-38f4-44a3-b986-8754c017e9b9 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 934b661f-3d18-4a3e-9fdc-ca3458be61cb] Setting instance to ERROR state.
2014-02-11 13:38:58.537 1161 WARNING nova.scheduler.driver [req-7a081f55-9e8d-41a1-9ce2-0a16c46831d6 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: bc63d821-f203-4344-8a90-f9c9760068ee] Setting instance to ERROR state.
2014-02-11 04:58:13.619 1161 WARNING nova.scheduler.driver [req-914688db-33e5-4c8c-8824-99caab91e7ba 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 34443dab-f1f9-4a00-bfb6-636b85bbbef4] Setting instance to ERROR state.
2014-02-11 05:32:45.815 1161 WARNING nova.scheduler.driver [req-1cbbd878-6358-4de5-a805-8017f15a4024 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: bd409f62-e7ce-4fb9-92a9-1a558c5fe396] Setting instance to ERROR state.
2014-02-11 14:35:39.835 1161 WARNING nova.scheduler.driver [req-ca8dc850-1385-4128-8f93-614a7c01641e 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 90f7c2cb-8044-4cc1-8f9f-a7e85ce35a51] Setting instance to ERROR state.
2014-02-11 14:37:41.206 1161 WARNING nova.scheduler.driver [req-d67abe4d-3b15-4aa3-a9d7-13f302c7d039 970ed56ef7bc41d59c54f5ed8a1690dc d0a0acfdb62b4cc8a2bfa8d6a08bb62f] [instance: 90ac3c0d-643c-479f-a1bc-6a126efc9a25] Setting instance to ERROR state.
All suspended instances of Ubuntu 13.10 and Fedora 20 may be resumed. But I am unable to create any new instance ( even Cirros) it immediately gets ERROR state
Second Cluster was done yesterday night and was alive just for several hours. First one was down. But this morning
it started behave the same way as second at about 2 a.m. , even before 'yum -y update'.
Same log on another controller :-
2014-02-11 03:06:48.050 767 ERROR nova.openstack.common.rpc.impl_qpid [req-05e0de62-9a83-47fe-b24d-1fbfd0d2b0a0 None None] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
2014-02-11 03:07:57.333 747 WARNING nova.openstack.common.db.sqlalchemy.session [req-556615f6-9274-45da-8135-9e40e73e5642 None None] SQL connection failed. infinite attempts left.
2014-02-11 03:09:47.979 747 WARNING nova.scheduler.driver [req-6f2a0473-7974-4cdb-8695-d9d170fce653 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 2dbebc5f-ead6-4aa1-93b1-bf66c5809959] Setting instance to ERROR state.
2014-02-11 03:16:20.056 747 WARNING nova.scheduler.driver [req-fee47a38-b5dc-450c-8fc5-865fd6155910 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: fa298251-10fe-43e1-b709-38bd47393bb4] Setting instance to ERROR state.
2014-02-11 03:18:19.799 745 WARNING nova.openstack.common.db.sqlalchemy.session [req-f1d2d706-0b26-401e-9e31-c4e23b583da4 None None] SQL connection failed. infinite attempts left.
2014-02-11 03:20:07.948 745 WARNING nova.scheduler.driver [req-a5d21042-0a31-47a5-a1d6-aff761bd4598 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 06e477e6-6eae-43b2-95d3-a4473f7add2e] Setting instance to ERROR state.
2014-02-11 11:42:32.059 762 WARNING nova.openstack.common.db.sqlalchemy.session [req-69ee79ff-3ffc-4acc-9d0e-67b99f061610 None None] SQL connection failed. infinite attempts left.
2014-02-11 12:47:17.438 762 WARNING nova.scheduler.driver [req-ba07c02f-cf9a-462d-a337-8df13e0c2926 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: db57afc6-e980-45fc-a8ba-fcb64699d4a4] Setting instance to ERROR state.
2014-02-11 12:55:20.508 762 WARNING nova.scheduler.driver [req-cc5b29e5-8819-4912-94db-46e9558ae65b 31614396b0444581a1f3c2d2e7c6f663 f2cc7bad8ac34b11908f5fa54b3205af] [instance: 6c3ec297-2bcd-4ea6-954a-d7aa0f20dbe6] Setting instance to ERROR state.
2014-02-11 17:47:17.533 751 WARNING nova.openstack.common.db.sqlalchemy.session [req-2cefe83c-4117-4212-825a-bf095617769f None None] SQL connection ...
View also https://bugzilla.redhat.com/show_bug....