Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Trove instance hangs in build status .

Openstack set up is packstack on centos 7 . rocky version of openstack . Enable trove using config_trove_install=y . Trove nova instance is up and running active status . Trove guest agent and database package is installed on trove nova instance through cloud-init via /etc/trove/cloudinit/mysql.cloudinit .

Expected output : Trove database instance to be active

Errors : Keystone log says POST http://172.16.51.84:35357/v3/auth/tokens ERROR keystone.assignment.core Circular reference found role inference rules

Trove-taskmanager log says Failed to send usage create-event for instance NotFound: Instance 16a4521c-ca95-4012-b566-be0ea2188baa could not be found. (HTTP 404)

Keystone log error

2019-03-12 15:25:34.752 13328 INFO keystone.common.wsgi [req-cdae11a1-5f04-47a4-94c1-ed96cc70b9b3 - - - - -] POST http://172.16.51.84:35357/v3/auth/tokens 2019-03-12 15:25:34.781 13327 INFO keystone.common.wsgi [req-01e354c1-eb45-4d19-a42f-1b9c623e4e68 - - - - -] POST http://172.16.51.84:35357/v3/auth/tokens 2019-03-12 15:25:35.128 13328 ERROR keystone.assignment.core [req-cdae11a1-5f04-47a4-94c1-ed96cc70b9b3 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.193 13328 ERROR keystone.assignment.core [req-cdae11a1-5f04-47a4-94c1-ed96cc70b9b3 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.203 13328 ERROR keystone.assignment.core [req-cdae11a1-5f04-47a4-94c1-ed96cc70b9b3 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.223 13327 ERROR keystone.assignment.core [req-01e354c1-eb45-4d19-a42f-1b9c623e4e68 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.290 13327 ERROR keystone.assignment.core [req-01e354c1-eb45-4d19-a42f-1b9c623e4e68 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.300 13327 ERROR keystone.assignment.core [req-01e354c1-eb45-4d19-a42f-1b9c623e4e68 - - - - -] Circular reference found role inference rules - cf202d4bfada47de94ed6052a34572ca. 2019-03-12 15:25:35.810 13325 INFO keystone.common.wsgi [req-61589e54-d906-4529-bd2a-14d5fe436b3d f9dfb0cbd3d04bf2be11f0e466b1921e 0f2babbe7b104b69bd8524747ff7e752 - default default] GET http://172.16.51.84:35357/v3/auth/tokens

trove-taskmanager.log

2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models [-] Failed to send usage create-event for instance 84b07540-8594-47bb-b0e6-36304699b3e1.: NotFound: Instance 16a4521c-ca95-4012-b566-be0ea2188baa could not be found. (HTTP 404) (Request-ID: req-effe4532-a841-4d56-a602-dc934a9213ed) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models Traceback (most recent call last): 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 439, in wait_for_instance 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models time_out=timeout) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/trove/common/utils.py", line 214, in poll_until 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models sleep_time=sleep_time, time_out=time_out).wait() 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models return hubs.get_hub().switch() 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models return self.greenlet.switch() 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 187, in _run_loop 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models result = func(self.args, *self.kw) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/trove/common/utils.py", line 192, in poll_and_check 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models obj = retriever() 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 712, in _service_is_active 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models server = self.nova_client.servers.get(c_id) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 785, in get 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models return self._get("/servers/%s" % base.getid(server), "server") 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 356, in _get 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models resp, body = self.api.client.get(url) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models return self.request(url, 'GET', **kwargs) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models raise exceptions.from_response(resp, body, url, method) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models NotFound: Instance 16a4521c-ca95-4012-b566-be0ea2188baa could not be found. (HTTP 404) (Request-ID: req-effe4532-a841-4d56-a602-dc934a9213ed) 2019-03-12 15:04:32.113 23468 ERROR trove.taskmanager.models

trove-troveapi.log

019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api Traceback (most recent call last): 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 90, in _call 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api result = cctxt.call(self.context, method_name, **kwargs) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api retry=self.retry) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api retry=retry) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api call_monitor_timeout, retry=retry) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api call_monitor_timeout) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api message = self.waiters.get(msg_id, timeout=timeout) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api 'to message ID %s' % msg_id) 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api MessagingTimeout: Timed out waiting for a reply to message ID d9003b3f26ff408bb584fe44e4202abc 2019-03-12 15:01:37.613 23515 ERROR trove.guestagent.api 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models [-] An error occurred communicating with the guest: Timed out waiting for a reply to message ID d9003b3f26ff408bb584fe44e4202abc.: GuestError: An error occurred communicating with the guest: Timed out waiting for a reply to message ID d9003b3f26ff408bb584fe44e4202abc. 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models Traceback (most recent call last): 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/instance/models.py", line 567, in load_guest_info 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models volume_info = guest.get_volume_info() 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 438, in get_volume_info 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models version=version, fs_path=None) 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 99, in _call 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models raise exception.GuestError(original_message=str(e)) 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models GuestError: An error occurred communicating with the guest: Timed out waiting for a reply to message ID d9003b3f26ff408bb584fe44e4202abc. 2019-03-12 15:01:37.785 23515 ERROR trove.instance.models 2019-03-12 15:01:38.310 23515 INFO eventlet.wsgi [-] 172.16.51.84 - - [12/Mar/2019 15:01:38] "GET /v1.0/15e11195c050411196028508022a494c/instances/84b07540-8594-47bb-b0e6-36304699b3e1 HTTP/1.1" 200 1081 6.633584 2019-03-12 15:01:38.644 23515 INFO eventlet.wsgi [-] 172.16.51.84 - - [12/Mar/2019 15:01:38] "GET /v1.0/15e11195c050411196028508022a494c/flavors/2 HTTP/1.1" 200 423 0.331612