Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

trove instance creation fail

I'm deploying three node architecture with Openstack Mitaka, and having trouble with Trove. I tried to make Trove instance, and instance's status remains in BUILD, and doesn't proceed from that point. After a long period of time, status turns to ERROR, showing this error on trove-taskmanager. I can't delete the instance until it turns to error. However on dashboard I see an ACTIVE instance on project>>compute>>instance tab, on project>database tab, it shows same status as trove-list command on controller:BUILD or ERROR. Is there any help on this issue? where should I be looking into associated with this problem?

trove list

+--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+ | ID | Name | Datastore | Datastore Version | Status | Flavor ID | Size | +--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+ | 0280f460-d3ba-44ca-9a6a-134449f7d4f8 | mysql_instance_4 | mysql | mysql-5.6 | BUILD | 6 | 5 | | 40cb9527-59f7-4cfe-997c-e23300bc6ba1 | test1 | mysql | mysql-5.6 | BUILD | 6 | 5 | | a92e31a8-54ed-445f-a153-17aeeb401b57 | test4 | mysql | mysql-5.6 | BUILD | 6 | 5 | +--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+

tail /var/log/trove/trove-taskmanage.log

2016-08-17 16:19:02.921 22971 DEBUG trove.guestagent.api [-] Sending the call to prepare the Guest. prepare /usr/lib/python2.7/dist-packages/trove/guestagent/api.
py:235
2016-08-17 16:19:02.954 22971 DEBUG trove.guestagent.api [-] Casting prepare _cast /usr/lib/python2.7/dist-packages/trove/guestagent/api.py:77
2016-08-17 16:19:02.955 22971 DEBUG oslo_messaging._drivers.amqpdriver [-] CAST unique_id: 1abf9a25ae044f90aa5df5267608f5e5 exchange 'trove' topic 'guestagent.db0
5cab4-555c-4566-a7f0-16b363a91652' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-08-17 16:19:02.957 22971 INFO trove.instance.models [-] Resetting task status to NONE on instance db05cab4-555c-4566-a7f0-16b363a91652.
2016-08-17 16:19:02.962 22971 DEBUG trove.db.models [-] Saving DBInstance: {u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'db05cab4-555c-45
66-a7f0-16b363a91652', u'datastore_version_id': u'38e939c7-98cd-47f7-a4f1-ee3efdadff67', 'errors': {}, u'hostname': None, u'server_status': None, u'task_descripti
on': 'No tasks for the instance.', u'volume_size': 5, u'type': None, u'updated': datetime.datetime(2016, 8, 17, 7, 19, 2, 962620), '_sa_instance_state': <sqlalche
my.orm.state.InstanceState object at 0x7fafc1276090>, u'deleted': 0, u'configuration_id': None, u'volume_id': u'321c1b65-cbd3-4009-afc2-7571da91d879', u'slave_of_
id': None, u'task_start_time': None, u'name': u'test2', u'task_id': 1, u'created': datetime.datetime(2016, 8, 17, 7, 18, 57), u'tenant_id': u'fda14487a68f436da7b5
5fe218798ea2', u'compute_instance_id': u'884975a2-4e48-4392-bc40-9e1f166e54bc', u'flavor_id': u'6'} save /usr/lib/python2.7/dist-packages/trove/db/models.py:62
2016-08-17 16:19:02.995 22971 DEBUG trove.taskmanager.models [-] trove dns support = False _create_dns_entry /usr/lib/python2.7/dist-packages/trove/taskmanager/mo
dels.py:942
2016-08-17 16:19:02.995 22971 DEBUG trove.taskmanager.models [-] <greenlet.greenlet object at 0x7fafc14367d0>: DNS not enabled for instance: db05cab4-555c-4566-a7
f0-16b363a91652 _create_dns_entry /usr/lib/python2.7/dist-packages/trove/taskmanager/models.py:983
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'trove.common.utils.poll_and_check' failed
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall Traceback (most recent call last):
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 136, in _run_loop
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/dist-packages/trove/common/utils.py", line 200, in poll_and_check
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall     raise exception.PollTimeOut
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall PollTimeOut: Polling request timed out.
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall 
2016-08-17 16:34:03.214 22971 ERROR trove.taskmanager.models [-] Failed to create instance db05cab4-555c-4566-a7f0-16b363a91652. Timeout waiting for instance to b
ecome active. No usage create-event was sent.
2016-08-17 16:34:03.252 22971 ERROR trove.taskmanager.models [-] Service status: ERROR
Service error description: guestagent error
2016-08-17 16:34:03.256 22971 DEBUG trove.db.models [-] Saving DBInstance: {u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'db05cab4-555c-45
66-a7f0-16b363a91652', u'datastore_version_id': u'38e939c7-98cd-47f7-a4f1-ee3efdadff67', 'errors': {}, u'hostname': None, u'server_status': None, u'task_descripti
on': 'Build error: guestagent timeout.', u'volume_size': 5, u'type': None, u'updated': datetime.datetime(2016, 8, 17, 7, 34, 3, 256637), '_sa_instance_state': <sq
lalchemy.orm.state.InstanceState object at 0x7fafc1276c50>, u'deleted': 0, u'configuration_id': None, u'volume_id': u'321c1b65-cbd3-4009-afc2-7571da91d879', u'sla
ve_of_id': None, u'task_start_time': None, u'name': u'test2', u'task_id': 84, u'created': datetime.datetime(2016, 8, 17, 7, 18, 57), u'tenant_id': u'fda14487a68f4
36da7b55fe218798ea2', u'compute_instance_id': u'884975a2-4e48-4392-bc40-9e1f166e54bc', u'flavor_id': u'6'} save /usr/lib/python2.7/dist-packages/trove/db/models.p
y:62
2016-08-17 16:34:03.268 22971 ERROR trove.taskmanager.models [-] Trove instance status: ERROR
Trove instance status description: Build error: guestagent timeout.
2016-08-17 16:40:22.360 22971 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None __call__ /usr/lib/python2.7/dist-packages/o
slo_messaging/_drivers/amqpdriver.py:201
2016-08-17 16:40:22.369 22971 DEBUG trove.instance.models [-] Instance 4a2f3780-d098-473d-9165-3baa2eb4d789 service status is new. load_instance /usr/lib/python2.
7/dist-packages/trove/instance/models.py:491
click to hide/show revision 2
No.2 Revision

trove instance creation fail

I'm deploying three node architecture with Openstack Mitaka, and having trouble with Trove. I tried to make Trove instance, and instance's status remains in BUILD, and doesn't proceed from that point. After a long period of time, status turns to ERROR, showing this error on trove-taskmanager. I can't delete the instance until it turns to error. However on dashboard I see an ACTIVE instance on project>>compute>>instance tab, on project>database tab, it shows same status as trove-list command on controller:BUILD or ERROR. Is there any help on this issue? where should I be looking into associated with this problem?

trove list

#trove list
+--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+
| ID                                   | Name             | Datastore | Datastore Version | Status | Flavor ID | Size |
+--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+
| 0280f460-d3ba-44ca-9a6a-134449f7d4f8 | mysql_instance_4 | mysql     | mysql-5.6         | BUILD  | 6         |    5 |
| 40cb9527-59f7-4cfe-997c-e23300bc6ba1 | test1            | mysql     | mysql-5.6         | BUILD  | 6         |    5 |
| a92e31a8-54ed-445f-a153-17aeeb401b57 | test4            | mysql     | mysql-5.6         | BUILD  | 6         |    5 |
+--------------------------------------+------------------+-----------+-------------------+--------+-----------+------++--------------------------------------+------------------+-----------+-------------------+--------+-----------+------+

#tail /var/log/trove/trove-taskmanage.log

tail /var/log/trove/trove-taskmanage.log

2016-08-17 16:19:02.921 22971 DEBUG trove.guestagent.api [-] Sending the call to prepare the Guest. prepare /usr/lib/python2.7/dist-packages/trove/guestagent/api.
py:235
2016-08-17 16:19:02.954 22971 DEBUG trove.guestagent.api [-] Casting prepare _cast /usr/lib/python2.7/dist-packages/trove/guestagent/api.py:77
2016-08-17 16:19:02.955 22971 DEBUG oslo_messaging._drivers.amqpdriver [-] CAST unique_id: 1abf9a25ae044f90aa5df5267608f5e5 exchange 'trove' topic 'guestagent.db0
5cab4-555c-4566-a7f0-16b363a91652' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-08-17 16:19:02.957 22971 INFO trove.instance.models [-] Resetting task status to NONE on instance db05cab4-555c-4566-a7f0-16b363a91652.
2016-08-17 16:19:02.962 22971 DEBUG trove.db.models [-] Saving DBInstance: {u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'db05cab4-555c-45
66-a7f0-16b363a91652', u'datastore_version_id': u'38e939c7-98cd-47f7-a4f1-ee3efdadff67', 'errors': {}, u'hostname': None, u'server_status': None, u'task_descripti
on': 'No tasks for the instance.', u'volume_size': 5, u'type': None, u'updated': datetime.datetime(2016, 8, 17, 7, 19, 2, 962620), '_sa_instance_state': <sqlalche
my.orm.state.InstanceState object at 0x7fafc1276090>, u'deleted': 0, u'configuration_id': None, u'volume_id': u'321c1b65-cbd3-4009-afc2-7571da91d879', u'slave_of_
id': None, u'task_start_time': None, u'name': u'test2', u'task_id': 1, u'created': datetime.datetime(2016, 8, 17, 7, 18, 57), u'tenant_id': u'fda14487a68f436da7b5
5fe218798ea2', u'compute_instance_id': u'884975a2-4e48-4392-bc40-9e1f166e54bc', u'flavor_id': u'6'} save /usr/lib/python2.7/dist-packages/trove/db/models.py:62
2016-08-17 16:19:02.995 22971 DEBUG trove.taskmanager.models [-] trove dns support = False _create_dns_entry /usr/lib/python2.7/dist-packages/trove/taskmanager/mo
dels.py:942
2016-08-17 16:19:02.995 22971 DEBUG trove.taskmanager.models [-] <greenlet.greenlet object at 0x7fafc14367d0>: DNS not enabled for instance: db05cab4-555c-4566-a7
f0-16b363a91652 _create_dns_entry /usr/lib/python2.7/dist-packages/trove/taskmanager/models.py:983
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'trove.common.utils.poll_and_check' failed
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall Traceback (most recent call last):
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 136, in _run_loop
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/dist-packages/trove/common/utils.py", line 200, in poll_and_check
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall     raise exception.PollTimeOut
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall PollTimeOut: Polling request timed out.
2016-08-17 16:34:03.213 22971 ERROR oslo.service.loopingcall 
2016-08-17 16:34:03.214 22971 ERROR trove.taskmanager.models [-] Failed to create instance db05cab4-555c-4566-a7f0-16b363a91652. Timeout waiting for instance to b
ecome active. No usage create-event was sent.
2016-08-17 16:34:03.252 22971 ERROR trove.taskmanager.models [-] Service status: ERROR
Service error description: guestagent error
2016-08-17 16:34:03.256 22971 DEBUG trove.db.models [-] Saving DBInstance: {u'cluster_id': None, u'shard_id': None, u'deleted_at': None, u'id': u'db05cab4-555c-45
66-a7f0-16b363a91652', u'datastore_version_id': u'38e939c7-98cd-47f7-a4f1-ee3efdadff67', 'errors': {}, u'hostname': None, u'server_status': None, u'task_descripti
on': 'Build error: guestagent timeout.', u'volume_size': 5, u'type': None, u'updated': datetime.datetime(2016, 8, 17, 7, 34, 3, 256637), '_sa_instance_state': <sq
lalchemy.orm.state.InstanceState object at 0x7fafc1276c50>, u'deleted': 0, u'configuration_id': None, u'volume_id': u'321c1b65-cbd3-4009-afc2-7571da91d879', u'sla
ve_of_id': None, u'task_start_time': None, u'name': u'test2', u'task_id': 84, u'created': datetime.datetime(2016, 8, 17, 7, 18, 57), u'tenant_id': u'fda14487a68f4
36da7b55fe218798ea2', u'compute_instance_id': u'884975a2-4e48-4392-bc40-9e1f166e54bc', u'flavor_id': u'6'} save /usr/lib/python2.7/dist-packages/trove/db/models.p
y:62
2016-08-17 16:34:03.268 22971 ERROR trove.taskmanager.models [-] Trove instance status: ERROR
Trove instance status description: Build error: guestagent timeout.
2016-08-17 16:40:22.360 22971 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None __call__ /usr/lib/python2.7/dist-packages/o
slo_messaging/_drivers/amqpdriver.py:201
2016-08-17 16:40:22.369 22971 DEBUG trove.instance.models [-] Instance 4a2f3780-d098-473d-9165-3baa2eb4d789 service status is new. load_instance /usr/lib/python2.
7/dist-packages/trove/instance/models.py:491