Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

asked 2017-02-21 07:26:34 -0600

seachina gravatar image

failed to volume-attach to a instance

I have finished openstack installation with Centos7.3.1611+newton exactly following the guide:https://docs.openstack.org/newton/install-guide-rdo/cinder.html. i can create the server and start from image and it's working, now i want attach a volume to the instance.

[root@controller ~]# nova list +--------------------------------------+----------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+----------------------+--------+------------+-------------+---------------------+ | 0d13f1ef-6c02-4688-ac0d-ee7e9fe48a95 | selfservice-instance | ACTIVE | - | Running | selfservice=3.3.3.5 | +--------------------------------------+----------------------+--------+------------+-------------+---------------------+ [root@controller ~]# cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | 32e01f2b-fda0-4245-a60a-6bb03bafa21b | available | test | 1 | - | false | | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ [root@controller ~]# nova volume-attach selfservice-instance 32e01f2b-fda0-4245-a60a-6bb03bafa21b /dev/vdb +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 32e01f2b-fda0-4245-a60a-6bb03bafa21b | | serverId | 0d13f1ef-6c02-4688-ac0d-ee7e9fe48a95 | | volumeId | 32e01f2b-fda0-4245-a60a-6bb03bafa21b | +----------+--------------------------------------+ [root@controller ~]# it's looks succesful without error, now i check the cloud disk status on storage, i did not found the target:

[root@storage1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert volume-32e01f2b-fda0-4245-a60a-6bb03bafa21b cinder-volumes -wi-a----- 1.00g
[root@storage1 ~]# targetcli targetcli shell version 2.1.fb41 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'.

/> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] />

Then i try to check the cinder logs in /var/log/cinder/api.log :

2017-02-21 20:35:28.459 15057 INFO cinder.api.openstack.wsgi [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes 2017-02-21 20:35:28.463 15057 INFO cinder.api.v2.volumes [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Create volume of 1 GB 2017-02-21 20:35:28.476 15057 INFO cinder.volume.api [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Availability Zones retrieved successfully. 2017-02-21 20:35:28.812 15057 INFO cinder.volume.api [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume created successfully. 2017-02-21 20:35:28.814 15057 INFO cinder.api.openstack.wsgi [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes returned with HTTP 202 2017-02-21 20:35:28.815 15057 INFO eventlet.wsgi.server [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes HTTP/1.1" status: 202 len: 1064 time: 0.9083891 2017-02-21 20:35:28.852 15057 INFO cinder.api.openstack.wsgi [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b 2017-02-21 20:35:28.920 15057 INFO cinder.volume.api [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:35:28.927 15057 INFO cinder.api.openstack.wsgi [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200 2017-02-21 20:35:28.928 15057 INFO eventlet.wsgi.server [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200 len: 1237 time: 0.1092041 2017-02-21 20:35:41.201 15049 INFO cinder.api.openstack.wsgi [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/detail 2017-02-21 20:35:41.275 15049 INFO cinder.volume.api [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Get all volumes completed successfully. 2017-02-21 20:35:41.282 15049 INFO cinder.api.openstack.wsgi [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/detail returned with HTTP 200 2017-02-21 20:35:41.284 15049 INFO eventlet.wsgi.server [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/detail HTTP/1.1" status: 200 len: 1280 time: 0.6968050 2017-02-21 20:36:15.559 15046 INFO cinder.api.openstack.wsgi [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b 2017-02-21 20:36:15.642 15046 INFO cinder.volume.api [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:15.649 15046 INFO cinder.api.openstack.wsgi [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200 2017-02-21 20:36:15.651 15046 INFO eventlet.wsgi.server [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200 len: 1277 time: 0.6712680 2017-02-21 20:36:15.693 15046 INFO cinder.api.openstack.wsgi [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action 2017-02-21 20:36:15.752 15046 INFO cinder.volume.api [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:15.772 15046 INFO cinder.volume.api [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Reserve volume completed successfully. 2017-02-21 20:36:15.773 15046 INFO cinder.api.openstack.wsgi [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 202 2017-02-21 20:36:15.775 15046 INFO eventlet.wsgi.server [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 202 len: 211 time: 0.1189210 2017-02-21 20:36:16.478 15062 INFO cinder.api.openstack.wsgi [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b 2017-02-21 20:36:16.562 15062 INFO cinder.volume.api [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:16.570 15062 INFO cinder.api.openstack.wsgi [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200 2017-02-21 20:36:16.571 15062 INFO eventlet.wsgi.server [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200 len: 1277 time: 0.7377150 2017-02-21 20:36:17.370 15062 INFO cinder.api.openstack.wsgi [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action 2017-02-21 20:36:17.445 15062 INFO cinder.volume.api [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Caught error: <type 'exceptions.oserror'=""> [Errno [Errno 13] Permission denied: '/var/lib/nova' Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming res = self.dispatcher.dispatch(message)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch result = func(ctxt, **new_args)

File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4400, in initialize_connection connector, volume=volume)

File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1414, in initialize_connection volume, connector)

File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 833, in create_export volume_path)

File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export **portals_config)

File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 127, in create_iscsi_target self._execute(*command_args, run_as_root=True)

File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 264, in inner do_log=False, semaphores=semaphores, delay=delay):

File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next()

File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 216, in lock ext_lock.acquire(delay=delay)

File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 151, in acquire self._do_open()

File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 115, in _do_open made_basedir = _ensure_tree(basedir)

File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 37, in _ensure_tree os.makedirs(path)

File "/usr/lib64/python2.7/os.py", line 150, in makedirs makedirs(head, mode)

File "/usr/lib64/python2.7/os.py", line 157, in makedirs mkdir(name, mode)

OSError: [Errno 13] Permission denied: '/var/lib/nova' ] Permission denied 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault Traceback (most recent call last): 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return req.get_response(self.application) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return request.get_response(self.application) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault response = req.get_response(self._app) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault response = self.app(environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__ 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault content_type, body, accept) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return method(req=request, *action_args) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/contrib/volume_actions.py", line 172, in _initialize_connection 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault connector) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 100, in wrapped 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return func(self, context, target_obj, *args, *kwargs) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 716, in initialize_connection 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault connector) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 244, in initialize_connection 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault return cctxt.call(ctxt, 'initialize_connection', *msg_args) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault retry=self.retry) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault timeout=timeout, retry=retry) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault retry=retry) 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault raise result 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault OSError: [Errno 13] Permission denied 2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault 2017-02-21 20:36:17.647 15062 INFO cinder.api.middleware.fault [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 500 2017-02-21 20:36:17.650 15062 INFO eventlet.wsgi.server [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 500 len: 425 time: 0.3421800 2017-02-21 20:36:17.691 15062 INFO cinder.api.openstack.wsgi [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action 2017-02-21 20:36:17.752 15062 INFO cinder.volume.api [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Caught error: <class 'oslo_messaging.rpc.client.remoteerror'=""> Remote error: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split' [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch\n result = func(ctxt, *new_args)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4404, in terminate_connection\n force=force)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1498, in terminate_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', u"VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'\n"]. None None. 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault Traceback (most recent call last): 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return req.get_response(self.application) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return request.get_response(self.application) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault response = req.get_response(self._app) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault response = self.app(environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault resp = self.call_func(req, args, *self.kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return self.func(req, args, *kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__ 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault content_type, body, accept) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return method(req=request, *action_args) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/contrib/volume_actions.py", line 194, in _terminate_connection 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault self.volume_api.terminate_connection(context, volume, connector) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 100, in wrapped 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault return func(self, context, target_obj, *args, *kwargs) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 726, in terminate_connection 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault force) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 250, in terminate_connection 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault connector=connector, force=force) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault retry=self.retry) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault timeout=timeout, retry=retry) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault retry=retry) 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault raise result 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault RemoteError: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split' 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4404, in terminate_connection\n force=force)\n', u' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1498, in terminate_connection\n raise exception.VolumeBackendAPIException(data=err_msg)\n', u"VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'\n"]. 2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault 2017-02-21 20:36:17.860 15062 INFO cinder.api.middleware.fault [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 500 2017-02-21 20:36:17.863 15062 INFO eventlet.wsgi.server [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 500 len: 425 time: 0.2052100 2017-02-21 20:36:17.983 15062 INFO cinder.api.openstack.wsgi [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action 2017-02-21 20:36:18.062 15062 INFO cinder.volume.api [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully. 2017-02-21 20:36:18.227 15062 INFO cinder.volume.api [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Unreserve volume completed successfully. 2017-02-21 20:36:18.228 15062 INFO cinder.api.openstack.wsgi [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 202 2017-02-21 20:36:18.229 15062 INFO eventlet.wsgi.server [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 202 len: 211 time: 0.3054399 [root@controller ~]#

The error "Permission denied: '/var/lib/nova'" appear in cinder ,i don't know why? anybody help me please?

click to hide/show revision 2
None

updated 2017-02-22 01:37:44 -0600

Bernd Bausch gravatar image

failed to volume-attach to a instance

I have finished openstack installation with Centos7.3.1611+newton exactly following the guide:https://docs.openstack.org/newton/install-guide-rdo/cinder.html. i can create the server and start from image and it's working, now i want attach a volume to the instance.

[root@controller ~]# nova list
+--------------------------------------+----------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                 | Status | Task State | Power State | Networks            |
+--------------------------------------+----------------------+--------+------------+-------------+---------------------+
| 0d13f1ef-6c02-4688-ac0d-ee7e9fe48a95 | selfservice-instance | ACTIVE | -          | Running     | selfservice=3.3.3.5 |
+--------------------------------------+----------------------+--------+------------+-------------+---------------------+
[root@controller ~]# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 32e01f2b-fda0-4245-a60a-6bb03bafa21b | available | test | 1    | -           | false    |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
[root@controller ~]# nova volume-attach selfservice-instance 32e01f2b-fda0-4245-a60a-6bb03bafa21b /dev/vdb
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 32e01f2b-fda0-4245-a60a-6bb03bafa21b |
| serverId | 0d13f1ef-6c02-4688-ac0d-ee7e9fe48a95 |
| volumeId | 32e01f2b-fda0-4245-a60a-6bb03bafa21b |
+----------+--------------------------------------+
[root@controller ~]# 
~]#

it's looks succesful without error, now i check the cloud disk status on storage, i did not found the target:

[root@storage1 ~]# lvs
  LV                                          VG             Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  volume-32e01f2b-fda0-4245-a60a-6bb03bafa21b cinder-volumes -wi-a----- 1.00g 
 
[root@storage1 ~]# targetcli
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
 
'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]
/> 
/>

Then i try to check the cinder logs in /var/log/cinder/api.log :

2017-02-21 20:35:28.459 15057 INFO cinder.api.openstack.wsgi [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes
2017-02-21 20:35:28.463 15057 INFO cinder.api.v2.volumes [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Create volume of 1 GB
2017-02-21 20:35:28.476 15057 INFO cinder.volume.api [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Availability Zones retrieved successfully.
2017-02-21 20:35:28.812 15057 INFO cinder.volume.api [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume created successfully.
2017-02-21 20:35:28.814 15057 INFO cinder.api.openstack.wsgi [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes returned with HTTP 202
2017-02-21 20:35:28.815 15057 INFO eventlet.wsgi.server [req-1ffca51d-18b5-4d47-ae42-3fe60fe919e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes HTTP/1.1" status: 202  len: 1064 time: 0.9083891
2017-02-21 20:35:28.852 15057 INFO cinder.api.openstack.wsgi [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b
2017-02-21 20:35:28.920 15057 INFO cinder.volume.api [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:35:28.927 15057 INFO cinder.api.openstack.wsgi [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200
2017-02-21 20:35:28.928 15057 INFO eventlet.wsgi.server [req-2192b1a0-cbcf-40b1-abd1-0ee4712fe9e0 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200  len: 1237 time: 0.1092041
2017-02-21 20:35:41.201 15049 INFO cinder.api.openstack.wsgi [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/detail
2017-02-21 20:35:41.275 15049 INFO cinder.volume.api [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Get all volumes completed successfully.
2017-02-21 20:35:41.282 15049 INFO cinder.api.openstack.wsgi [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/detail returned with HTTP 200
2017-02-21 20:35:41.284 15049 INFO eventlet.wsgi.server [req-29a8fa7b-fe92-4035-98fd-0f9ab6c3fad9 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/detail HTTP/1.1" status: 200  len: 1280 time: 0.6968050
2017-02-21 20:36:15.559 15046 INFO cinder.api.openstack.wsgi [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b
2017-02-21 20:36:15.642 15046 INFO cinder.volume.api [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:15.649 15046 INFO cinder.api.openstack.wsgi [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200
2017-02-21 20:36:15.651 15046 INFO eventlet.wsgi.server [req-22444c7e-6600-4f66-8a38-53e9b702ac74 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200  len: 1277 time: 0.6712680
2017-02-21 20:36:15.693 15046 INFO cinder.api.openstack.wsgi [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action
2017-02-21 20:36:15.752 15046 INFO cinder.volume.api [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:15.772 15046 INFO cinder.volume.api [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Reserve volume completed successfully.
2017-02-21 20:36:15.773 15046 INFO cinder.api.openstack.wsgi [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 202
2017-02-21 20:36:15.775 15046 INFO eventlet.wsgi.server [req-419859f1-1c43-4cb4-9972-c6fb55cc62d6 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.5 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 202  len: 211 time: 0.1189210
2017-02-21 20:36:16.478 15062 INFO cinder.api.openstack.wsgi [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] GET http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b
2017-02-21 20:36:16.562 15062 INFO cinder.volume.api [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:16.570 15062 INFO cinder.api.openstack.wsgi [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b returned with HTTP 200
2017-02-21 20:36:16.571 15062 INFO eventlet.wsgi.server [req-fef18c83-1dd7-4c80-80fd-135cc27eb182 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "GET /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b HTTP/1.1" status: 200  len: 1277 time: 0.7377150
2017-02-21 20:36:17.370 15062 INFO cinder.api.openstack.wsgi [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action
2017-02-21 20:36:17.445 15062 INFO cinder.volume.api [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Caught error: <type 'exceptions.oserror'=""> 'exceptions.OSError'> [Errno [Errno 13] Permission denied: '/var/lib/nova'
Traceback (most recent call last):
 
last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
    res = self.dispatcher.dispatch(message)
 
self.dispatcher.dispatch(message)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
 
args)

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
    result = func(ctxt, **new_args)
 
**new_args)

  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4400, in initialize_connection
    connector, volume=volume)
 
volume=volume)

  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1414, in initialize_connection
    volume, connector)
 
connector)

  File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 833, in create_export
    volume_path)
 
volume_path)

  File "/usr/lib/python2.7/site-packages/cinder/volume/targets/iscsi.py", line 210, in create_export
    **portals_config)
 
**portals_config)

  File "/usr/lib/python2.7/site-packages/cinder/volume/targets/lio.py", line 127, in create_iscsi_target
    self._execute(*command_args, run_as_root=True)
 
run_as_root=True)

  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 264, in inner
    do_log=False, semaphores=semaphores, delay=delay):
 
delay=delay):

  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
 
self.gen.next()

  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 216, in lock
    ext_lock.acquire(delay=delay)
 
ext_lock.acquire(delay=delay)

  File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 151, in acquire
    self._do_open()
 
self._do_open()

  File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 115, in _do_open
    made_basedir = _ensure_tree(basedir)
 
_ensure_tree(basedir)

  File "/usr/lib/python2.7/site-packages/fasteners/process_lock.py", line 37, in _ensure_tree
    os.makedirs(path)
 
os.makedirs(path)

  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
 
mode)

  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
 
mode)

OSError: [Errno 13] Permission denied: '/var/lib/nova'
] Permission denied
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return request.get_response(self.application)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     content_type, body, accept)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     action_result = self.dispatch(meth, request, action_args)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return method(req=request, *action_args)
**action_args)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/contrib/volume_actions.py", line 172, in _initialize_connection
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     connector)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 100, in wrapped
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return func(self, context, target_obj, *args, *kwargs)
**kwargs)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 716, in initialize_connection
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     connector)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 244, in initialize_connection
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     return cctxt.call(ctxt, 'initialize_connection', *msg_args)
**msg_args)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     retry=self.retry)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     timeout=timeout, retry=retry)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     retry=retry)
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault     raise result
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault OSError: [Errno 13] Permission denied
2017-02-21 20:36:17.636 15062 ERROR cinder.api.middleware.fault 
2017-02-21 20:36:17.647 15062 INFO cinder.api.middleware.fault [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 500
2017-02-21 20:36:17.650 15062 INFO eventlet.wsgi.server [req-0e14c5e0-4485-4474-82be-c8fc73f220f3 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 500  len: 425 time: 0.3421800
2017-02-21 20:36:17.691 15062 INFO cinder.api.openstack.wsgi [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action
2017-02-21 20:36:17.752 15062 INFO cinder.volume.api [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Caught error: <class 'oslo_messaging.rpc.client.remoteerror'=""> 'oslo_messaging.rpc.client.RemoteError'> Remote error: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'
[u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch\n    result = func(ctxt, *new_args)\n', **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4404, in terminate_connection\n    force=force)\n', u'  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1498, in terminate_connection\n    raise exception.VolumeBackendAPIException(data=err_msg)\n', u"VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'\n"]. None
None.
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return request.get_response(self.application)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     resp = self.call_func(req, args, *self.kwargs)
*args, **self.kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return self.func(req, args, *kwargs)
*args, **kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     content_type, body, accept)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     action_result = self.dispatch(meth, request, action_args)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return method(req=request, *action_args)
**action_args)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/contrib/volume_actions.py", line 194, in _terminate_connection
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     self.volume_api.terminate_connection(context, volume, connector)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 100, in wrapped
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     return func(self, context, target_obj, *args, *kwargs)
**kwargs)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/api.py", line 726, in terminate_connection
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     force)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 250, in terminate_connection
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     connector=connector, force=force)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     retry=self.retry)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     timeout=timeout, retry=retry)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     retry=retry)
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault     raise result
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault RemoteError: Remote error: VolumeBackendAPIException Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 4404, in terminate_connection\n    force=force)\n', u'  File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1498, in terminate_connection\n    raise exception.VolumeBackendAPIException(data=err_msg)\n', u"VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Terminate volume connection failed: 'NoneType' object has no attribute 'split'\n"].
2017-02-21 20:36:17.853 15062 ERROR cinder.api.middleware.fault 
2017-02-21 20:36:17.860 15062 INFO cinder.api.middleware.fault [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 500
2017-02-21 20:36:17.863 15062 INFO eventlet.wsgi.server [req-27af3e32-34da-41de-81c9-1f9134df228b 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: 500  len: 425 time: 0.2052100
2017-02-21 20:36:17.983 15062 INFO cinder.api.openstack.wsgi [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] POST http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action
2017-02-21 20:36:18.062 15062 INFO cinder.volume.api [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Volume info retrieved successfully.
2017-02-21 20:36:18.227 15062 INFO cinder.volume.api [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] Unreserve volume completed successfully.
2017-02-21 20:36:18.228 15062 INFO cinder.api.openstack.wsgi [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] http://controller:8776/v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action returned with HTTP 202
2017-02-21 20:36:18.229 15062 INFO eventlet.wsgi.server [req-a9e6e518-668a-4263-a3dd-8f24a432f516 090c13cc34ef44d9b1c44f9ca588fa2f d512da13581f43dd8bf42a525afb830f - default default] 1.1.1.6 "POST /v2/d512da13581f43dd8bf42a525afb830f/volumes/32e01f2b-fda0-4245-a60a-6bb03bafa21b/action HTTP/1.1" status: status

: 202  len: 211 time: 0.3054399
[root@controller ~]# 
~]#

The error "Permission denied: '/var/lib/nova'" appear in cinder ,i don't know why? anybody help me please?