cinder nfs snapshot create and delete fail (No JSON object could be decoded)
Hello!
I have a "little" problem. :(
I can not create snapshot on nfs backend, and I can not delete the failed snapshot.
So. This is openstack rocky. I can create and delete volumes on non nfs backend and I can create and delete snapshots too. I can create and delete volumes on nfs backend and I get error if I want to create snapshot on it.
Please help me.
# openstack volume snapshot create --volume 830ed757-11d6-44d1-a010-602fc245a930 snapshot1
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2019-07-24T14:25:30.369324 |
| description | None |
| id | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name | snapshot1 |
| properties | |
| size | 40 |
| status | creating |
| updated_at | None |
| volume_id | 830ed757-11d6-44d1-a010-602fc245a930 |
+-------------+--------------------------------------+
# openstack volume snapshot list --all-project
+--------------------------------------+----------------+-------------+----------------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+----------------+-------------+----------------+------+
| dea45c5c-1989-46bc-9587-1e0c55c9b37d | snapshot1 | None | error | 40 |
| cf61f910-6357-4e91-8be1-58fe8200be26 | snap1 | | error_deleting | 40 |
| 7c2eba49-25a6-432b-bc3d-e784265ace64 | rhel6-s | | error_deleting | 25 |
| 92d22714-fb08-4569-9b64-a6a5954bd224 | odma1-s | | error_deleting | 40 |
| adf6b7d9-34b2-4c53-b05b-8a68c0c1312a | odma1-snapshot | | error_deleting | 40 |
+--------------------------------------+----------------+-------------+----------------+------+
# openstack volume snapshot show dea45c5c-1989-46bc-9587-1e0c55c9b37d
+--------------------------------------------+--------------------------------------+
| Field | Value |
+--------------------------------------------+--------------------------------------+
| created_at | 2019-07-24T14:25:30.000000 |
| description | None |
| id | dea45c5c-1989-46bc-9587-1e0c55c9b37d |
| name | snapshot1 |
| os-extended-snapshot-attributes:progress | 0% |
| os-extended-snapshot-attributes:project_id | 324e6f1090d840b5877a1f706a3c839a |
| properties | |
| size | 40 |
| status | error |
| updated_at | 2019-07-24T14:25:30.000000 |
| volume_id | 830ed757-11d6-44d1-a010-602fc245a930 |
+--------------------------------------------+--------------------------------------+
2019-07-24 16:25:30.505 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" acquired by "create_snapshot" :: waited 0.000s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:150
2019-07-24 16:25:30.505 86676 DEBUG cinder.volume.drivers.remotefs [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Creating offline snapshot dea45c5c-1989-46bc-9587-1e0c55c9b37d of volume 830ed757-11d6-44d1-a010-602fc245a930 _create_snapshot /usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py:1399
2019-07-24 16:25:30.523 86676 DEBUG cinder.coordination [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Lock "/var/lib/cinder/cinder-nfs-830ed757-11d6-44d1-a010-602fc245a930" released by "create_snapshot" :: held 0.018s _synchronized /usr/lib/python2.7/site-packages/cinder/coordination.py:162
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server [req-3641b0ec-38a3-46f4-8289-dfd30211c1a1 3c98f1ce653d47fd94a60f43f6a96ac8 324e6f1090d840b5877a1f706a3c839a - default default] Exception during message handling: ValueError: No JSON object could be decoded
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_snapshot
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2019-07-24 16:25:30.534 86676 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder ...