Revision history [back]

click to hide/show revision 1
initial version

cinder nfs snapshot create and delete fail (No JSON object could be decoded)

Hello!

I have a "little" problem. :(

I can not create snapshot on nfs backend, and I can not delete the failed snapshot.

Please help me.

# openstack volume snapshot create --volume 830ed757-11d6-44d1-a010-602fc245a930 snapshot1
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| created_at  | 2019-07-24T14:25:30.369324           |
| description | None                                 |
| id          | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name        | snapshot1                            |
| properties  |                                      |
| size        | 40                                   |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 830ed757-11d6-44d1-a010-602fc245a930 |
+-------------+--------------------------------------+
# openstack volume snapshot list --all-project
+--------------------------------------+----------------+-------------+----------------+------+
| ID                                   | Name           | Description | Status         | Size |
+--------------------------------------+----------------+-------------+----------------+------+
| dea45c5c-1989-46bc-9587-1e0c55c9b37d | snapshot1      | None        | error          |   40 |
| cf61f910-6357-4e91-8be1-58fe8200be26 | snap1          |             | error_deleting |   40 |
| 7c2eba49-25a6-432b-bc3d-e784265ace64 | rhel6-s        |             | error_deleting |   25 |
| 92d22714-fb08-4569-9b64-a6a5954bd224 | odma1-s        |             | error_deleting |   40 |
| adf6b7d9-34b2-4c53-b05b-8a68c0c1312a | odma1-snapshot |             | error_deleting |   40 |
+--------------------------------------+----------------+-------------+----------------+------+
# openstack volume snapshot show dea45c5c-1989-46bc-9587-1e0c55c9b37d
+--------------------------------------------+--------------------------------------+
| Field                                      | Value                                |
+--------------------------------------------+--------------------------------------+
| created_at                                 | 2019-07-24T14:25:30.000000           |
| description                                | None                                 |
| id                                         | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name                                       | snapshot1                            |
| os-extended-snapshot-attributes:progress   | 0%                                   |
| os-extended-snapshot-attributes:project_id | 324e6f1090d840b5877a1f706a3c839a     |
| properties                                 |                                      |
| size                                       | 40                                   |
| status                                     | error                                |
| updated_at                                 | 2019-07-24T14:25:30.000000           |
| volume_id                                  | 830ed757-11d6-44d1-a010-602fc245a930 |
+--------------------------------------------+--------------------------------------+


2019-07-24 16:25:30.505 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" acquired by "create_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:25:30.505 86676 DEBUG cinder.volume.drivers.remotefs [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Creating offline snapshot dea45c5c-1989-46bc-9587-1e0c55c9b37d of volume 830ed757-11d6-44d1-a010-602fc245a930 _create_snapshot /usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py:1399
2019-07-24 16:25:30.523 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" released by "create_snapshot" :: held 0.018s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Exception during message handling: ValueError: No JSON object could be decoded
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1072, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     snapshot.save()
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1064, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     model_update = self.driver.create_snapshot(snapshot)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 566, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return self._create_snapshot(snapshot)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 1413, in _create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     snap_info = self._read_info_file(info_path, empty_if_missing=True)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 821, in _read_info_file
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return json.loads(self._read_file(info_path))
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return _default_decoder.decode(s)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     raise ValueError("No JSON object could be decoded")
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server ValueError: No JSON object could be decoded
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server


# openstack volume snapshot delete --force dea45c5c-1989-46bc-9587-1e0c55c9b37d
# openstack volume snapshot list --all-project
+--------------------------------------+----------------+-------------+----------------+------+
| ID                                   | Name           | Description | Status         | Size |
+--------------------------------------+----------------+-------------+----------------+------+
| dea45c5c-1989-46bc-9587-1e0c55c9b37d | snapshot1      | None        | error_deleting |   40 |
| cf61f910-6357-4e91-8be1-58fe8200be26 | snap1          |             | error_deleting |   40 |
| 7c2eba49-25a6-432b-bc3d-e784265ace64 | rhel6-s        |             | error_deleting |   25 |
| 92d22714-fb08-4569-9b64-a6a5954bd224 | odma1-s        |             | error_deleting |   40 |
| adf6b7d9-34b2-4c53-b05b-8a68c0c1312a | odma1-snapshot |             | error_deleting |   40 |
+--------------------------------------+----------------+-------------+----------------+------+

2019-07-24 16:28:49.941 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-dea45c5c-1989-46bc-9587-1e0c55c9b37d-delete_snapshot" acquired by "delete_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:28:49.942 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" acquired by "delete_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:28:49.943 86676 DEBUG cinder.volume.drivers.remotefs [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Deleting offline snapshot dea45c5c-1989-46bc-9587-1e0c55c9b37d of volume 830ed757-11d6-44d1-a010-602fc245a930 _delete_snapshot /usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py:1090
2019-07-24 16:28:49.960 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" released by "delete_snapshot" :: held 0.017s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:28:50.096 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-dea45c5c-1989-46bc-9587-1e0c55c9b37d-delete_snapshot" released by "delete_snapshot" :: held 0.155s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Exception during message handling: ValueError: No JSON object could be decoded
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1142, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     snapshot.save()
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1132, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     self.driver.delete_snapshot(snapshot)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 573, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return self._delete_snapshot(snapshot)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 1102, in _delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     snap_info = self._read_info_file(info_path, empty_if_missing=True)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 821, in _read_info_file
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return json.loads(self._read_file(info_path))
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return _default_decoder.decode(s)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     raise ValueError("No JSON object could be decoded")
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server ValueError: No JSON object could be decoded
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server

cinder nfs snapshot create and delete fail (No JSON object could be decoded)

Hello!

I have a "little" problem. :(

I can not create snapshot on nfs backend, and I can not delete the failed snapshot.

So. This is openstack rocky. I can create and delete volumes on non nfs backend and I can create and delete snapshots too. I can create and delete volumes on nfs backend and I get error if I want to create snapshot on it.

Please help me.

# openstack volume snapshot create --volume 830ed757-11d6-44d1-a010-602fc245a930 snapshot1
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| created_at  | 2019-07-24T14:25:30.369324           |
| description | None                                 |
| id          | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name        | snapshot1                            |
| properties  |                                      |
| size        | 40                                   |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 830ed757-11d6-44d1-a010-602fc245a930 |
+-------------+--------------------------------------+
# openstack volume snapshot list --all-project
+--------------------------------------+----------------+-------------+----------------+------+
| ID                                   | Name           | Description | Status         | Size |
+--------------------------------------+----------------+-------------+----------------+------+
| dea45c5c-1989-46bc-9587-1e0c55c9b37d | snapshot1      | None        | error          |   40 |
| cf61f910-6357-4e91-8be1-58fe8200be26 | snap1          |             | error_deleting |   40 |
| 7c2eba49-25a6-432b-bc3d-e784265ace64 | rhel6-s        |             | error_deleting |   25 |
| 92d22714-fb08-4569-9b64-a6a5954bd224 | odma1-s        |             | error_deleting |   40 |
| adf6b7d9-34b2-4c53-b05b-8a68c0c1312a | odma1-snapshot |             | error_deleting |   40 |
+--------------------------------------+----------------+-------------+----------------+------+
# openstack volume snapshot show dea45c5c-1989-46bc-9587-1e0c55c9b37d
+--------------------------------------------+--------------------------------------+
| Field                                      | Value                                |
+--------------------------------------------+--------------------------------------+
| created_at                                 | 2019-07-24T14:25:30.000000           |
| description                                | None                                 |
| id                                         | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name                                       | snapshot1                            |
| os-extended-snapshot-attributes:progress   | 0%                                   |
| os-extended-snapshot-attributes:project_id | 324e6f1090d840b5877a1f706a3c839a     |
| properties                                 |                                      |
| size                                       | 40                                   |
| status                                     | error                                |
| updated_at                                 | 2019-07-24T14:25:30.000000           |
| volume_id                                  | 830ed757-11d6-44d1-a010-602fc245a930 |
+--------------------------------------------+--------------------------------------+


2019-07-24 16:25:30.505 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" acquired by "create_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:25:30.505 86676 DEBUG cinder.volume.drivers.remotefs [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Creating offline snapshot dea45c5c-1989-46bc-9587-1e0c55c9b37d of volume 830ed757-11d6-44d1-a010-602fc245a930 _create_snapshot /usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py:1399
2019-07-24 16:25:30.523 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" released by "create_snapshot" :: held 0.018s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Exception during message handling: ValueError: No JSON object could be decoded
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1072, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     snapshot.save()
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1064, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     model_update = self.driver.create_snapshot(snapshot)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 566, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return self._create_snapshot(snapshot)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 1413, in _create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     snap_info = self._read_info_file(info_path, empty_if_missing=True)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 821, in _read_info_file
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return json.loads(self._read_file(info_path))
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     return _default_decoder.decode(s)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server     raise ValueError("No JSON object could be decoded")
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server ValueError: No JSON object could be decoded
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server


# openstack volume snapshot delete --force dea45c5c-1989-46bc-9587-1e0c55c9b37d
# openstack volume snapshot list --all-project
+--------------------------------------+----------------+-------------+----------------+------+
| ID                                   | Name           | Description | Status         | Size |
+--------------------------------------+----------------+-------------+----------------+------+
| dea45c5c-1989-46bc-9587-1e0c55c9b37d | snapshot1      | None        | error_deleting |   40 |
| cf61f910-6357-4e91-8be1-58fe8200be26 | snap1          |             | error_deleting |   40 |
| 7c2eba49-25a6-432b-bc3d-e784265ace64 | rhel6-s        |             | error_deleting |   25 |
| 92d22714-fb08-4569-9b64-a6a5954bd224 | odma1-s        |             | error_deleting |   40 |
| adf6b7d9-34b2-4c53-b05b-8a68c0c1312a | odma1-snapshot |             | error_deleting |   40 |
+--------------------------------------+----------------+-------------+----------------+------+

2019-07-24 16:28:49.941 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-dea45c5c-1989-46bc-9587-1e0c55c9b37d-delete_snapshot" acquired by "delete_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:28:49.942 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" acquired by "delete_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:28:49.943 86676 DEBUG cinder.volume.drivers.remotefs [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Deleting offline snapshot dea45c5c-1989-46bc-9587-1e0c55c9b37d of volume 830ed757-11d6-44d1-a010-602fc245a930 _delete_snapshot /usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py:1090
2019-07-24 16:28:49.960 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" released by "delete_snapshot" :: held 0.017s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:28:50.096 86676 DEBUG cinder.coordination [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-dea45c5c-1989-46bc-9587-1e0c55c9b37d-delete_snapshot" released by "delete_snapshot" :: held 0.155s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server [req-def84428-b0cb-4442-aadb-a70a2c0d181a 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Exception during message handling: ValueError: No JSON object could be decoded
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1142, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     snapshot.save()
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     self.force_reraise()
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1132, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     self.driver.delete_snapshot(snapshot)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 573, in delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return self._delete_snapshot(snapshot)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 1102, in _delete_snapshot
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     snap_info = self._read_info_file(info_path, empty_if_missing=True)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py", line 821, in _read_info_file
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return json.loads(self._read_file(info_path))
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     return _default_decoder.decode(s)
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server     raise ValueError("No JSON object could be decoded")
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server ValueError: No JSON object could be decoded
2019-07-24 16:28:50.097 86676 ERROR oslo_messaging.rpc.server