Ask Your Question

Revision history [back]

Log shows as below

openstack volume backup show 6eb7d456-ff45-471f-845a-cb50d652b014 +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | availability_zone | nova | | container | None | | created_at | 2018-10-30T08:57:44.000000 | | data_timestamp | 2018-10-28T17:00:58.000000 | | description | None | | fail_reason | Timed out waiting for a reply to message ID 5cad4be64ed84cb7aaad6882f379f5ea | | has_dependent_backups | False | | id | 6eb7d456-ff45-471f-845a-cb50d652b014 | | is_incremental | False | | name | backup03 | | object_count | 0 | | size | 20 | | snapshot_id | f958dd94-5c92-4786-9727-5ba44b06d927 | | status | error | | updated_at | 2018-10-30T09:34:50.000000 | | volume_id | 07f0c2e1-d16e-4ecb-919d-e2d8c507c6c2

2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server [req-b637e9b5-aed9-4659-808e-7b9bcca266b5 d76851a39ade402f9fca72371e84cc 07 be3672f49805466fb76ee0673280d5be - default default] Exception during message handling 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.p y", line 222, in dispatch 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.p y", line 192, in _do_dispatch 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 403, in create_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 2 20, in __exit__ 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 1 96, in force_reraise 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 397, in create_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self._run_backup(context, backup, volume) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 430, in _run_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server volume) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/rpcapi.py", line 355, in get_backup_device 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server want_objects=True) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server retry=self.retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", l ine 97, in _send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 458, in send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server retry=retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 447, in _send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server result = self._waiter.wait(msg_id, timeout) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 339, in wait 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server message = self.waiters.get(msg_id, timeout=timeout) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 238, in get 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server 'to message ID %s' % msg_id) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server MessagingTimeout: Timed out waiting for a reply to message ID 5cad4be64ed84cb7aaad6882f379f5ea

openstack volume service list +------------------+--------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+--------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | fgsusmioshyp3-prod | nova | enabled | up | 2018-10-30T11:20:04.000000 | | cinder-scheduler | fgsusmioshyp1-prod | nova | enabled | up | 2018-10-30T11:20:03.000000 | | cinder-scheduler | fgsusmioshyp2-prod | nova | enabled | up | 2018-10-30T11:20:05.000000 | | cinder-volume | fgsusmioshyp1-prod@unity | nova | enabled | up | 2018-10-30T11:20:07.000000 | | cinder-volume | fgsusmioshyp2-prod@unity | nova | enabled | up | 2018-10-30T11:20:03.000000 | | cinder-volume | fgsusmioshyp3-prod@unity | nova | enabled | up | 2018-10-30T11:20:10.000000 | | cinder-backup | fgsusmioshyp1-prod | nova | enabled | up | 2018-10-30T11:20:08.000000 | | cinder-backup | fgsusmioshyp2-prod | nova | enabled | up | 2018-10-30T11:20:08.000000 | | cinder-backup | fgsusmioshyp3-prod | nova | enabled | up | 2018-10-30T11:20:09.000000 | +------------------+--------------------------+------+---------+-------+----------------------------+

Log shows as below

openstack volume backup show 6eb7d456-ff45-471f-845a-cb50d652b014 6eb7d456-ff45-471f-845a-cb50d652b014

+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| availability_zone     | nova                                                                         |
| container             | None                                                                         |
| created_at            | 2018-10-30T08:57:44.000000                                                   |
| data_timestamp        | 2018-10-28T17:00:58.000000                                                   |
| description           | None                                                                         |
| fail_reason           | Timed **Timed out waiting for a reply to message ID 5cad4be64ed84cb7aaad6882f379f5ea 5cad4be64ed84cb7aaad6882f379f5ea** |
| has_dependent_backups | False                                                                        |
| id                    | 6eb7d456-ff45-471f-845a-cb50d652b014                                         |
| is_incremental        | False                                                                        |
| name                  | backup03                                                                     |
| object_count          | 0                                                                            |
| size                  | 20                                                                           |
| snapshot_id           | f958dd94-5c92-4786-9727-5ba44b06d927                                         |
| status                | error                                                                        |
| updated_at            | 2018-10-30T09:34:50.000000                                                   |
 | volume_id             | 07f0c2e1-d16e-4ecb-919d-e2d8c507c6c2

07f0c2e1-d16e-4ecb-919d-e2d8c507c6c2

2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server [req-b637e9b5-aed9-4659-808e-7b9bcca266b5 d76851a39ade402f9fca72371e84cc 07 be3672f49805466fb76ee0673280d5be - default default] Exception during message handling 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.p y", line 222, in dispatch 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.p y", line 192, in _do_dispatch 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 403, in create_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 2 20, in __exit__ 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 1 96, in force_reraise 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 397, in create_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server self._run_backup(context, backup, volume) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 430, in _run_backup 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server volume) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/volume/rpcapi.py", line 355, in get_backup_device 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server want_objects=True) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server retry=self.retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", l ine 97, in _send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 458, in send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server retry=retry) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 447, in _send 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server result = self._waiter.wait(msg_id, timeout) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 339, in wait 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server message = self.waiters.get(msg_id, timeout=timeout) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdri ver.py", line 238, in get 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server 'to message ID %s' % msg_id) 2018-10-30 08:58:44.763 1020448 ERROR oslo_messaging.rpc.server MessagingTimeout: Timed out waiting for a reply to message ID 5cad4be64ed84cb7aaad6882f379f5ea

openstack volume service list
+------------------+--------------------------+------+---------+-------+----------------------------+
| Binary           | Host                     | Zone | Status  | State | Updated At                 |
+------------------+--------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | fgsusmioshyp3-prod       | nova | enabled | up    | 2018-10-30T11:20:04.000000 |
| cinder-scheduler | fgsusmioshyp1-prod       | nova | enabled | up    | 2018-10-30T11:20:03.000000 |
| cinder-scheduler | fgsusmioshyp2-prod       | nova | enabled | up    | 2018-10-30T11:20:05.000000 |
| cinder-volume    | fgsusmioshyp1-prod@unity | nova | enabled | up    | 2018-10-30T11:20:07.000000 |
| cinder-volume    | fgsusmioshyp2-prod@unity | nova | enabled | up    | 2018-10-30T11:20:03.000000 |
| cinder-volume    | fgsusmioshyp3-prod@unity | nova | enabled | up    | 2018-10-30T11:20:10.000000 |
| cinder-backup    | fgsusmioshyp1-prod       | nova | enabled | up    | 2018-10-30T11:20:08.000000 |
| cinder-backup    | fgsusmioshyp2-prod       | nova | enabled | up    | 2018-10-30T11:20:08.000000 |
| cinder-backup    | fgsusmioshyp3-prod       | nova | enabled | up    | 2018-10-30T11:20:09.000000 |

+------------------+--------------------------+------+---------+-------+----------------------------+