Hi, I use packstack to install Openstack with one controller node and two compute node on F20. I'm sure all packets are updated. Instances can start normally without volumes, but fail to spawn when any volume is attached. After go over the nova log, I found nova can not connect the iscsi target. Further debug show there is no target exposed on controller node. Then I found the following in cinder volume.log. I don't know what to do next. Google gives no luck too. (I tried to edit cinder.conf to change "iscsi_helper=lioadm" to "iscsi_helper=lioadm", which doesn't help) Is this a Bug or is there something I messed up ?
2014-12-17 20:10:42.324 1375 INFO cinder.image.image_utils [req-e9008623-7d2f-4dc6-9f0c-623f54b4e33d 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Image download 12.00 MB at 12.00 MB/s 2014-12-17 20:10:43.571 1375 INFO cinder.image.image_utils [req-e9008623-7d2f-4dc6-9f0c-623f54b4e33d 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Converted 12.00 MB image at 12.00 MB/s 2014-12-17 20:10:44.559 1375 INFO cinder.volume.flows.manager.create_volume [req-e9008623-7d2f-4dc6-9f0c-623f54b4e33d 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Volume volume-eacfef77-1a5f-49e0-a13f-786d0d58c214 (eacfef77-1a5f-49e0-a13f-786d0d58c214): created successfully 2014-12-17 20:10:45.646 1375 INFO cinder.brick.iscsi.iscsi [req-f1c65d49-ab3c-41f4-9764-3e6321a7c6cd 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Creating iscsi_target for volume: volume-eacfef77-1a5f-49e0-a13f-786d0d58c214 2014-12-17 20:10:46.423 1375 ERROR cinder.brick.iscsi.iscsi [req-f1c65d49-ab3c-41f4-9764-3e6321a7c6cd 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Failed to create iscsi target for volume id:volume-eacfef77-1a5f-49e0-a13f-786d0d58c214. 2014-12-17 20:10:46.426 1375 ERROR oslo.messaging.rpc.dispatcher [req-f1c65d49-ab3c-41f4-9764-3e6321a7c6cd 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Exception during message handling: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, new_args) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher return f(args, *kwargs) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 875, in initialize_connection 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher volume) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher return f(args, *kwargs) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 548, in create_export 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher return self._create_export(context, volume) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 560, in _create_export 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher self.configuration) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 61, in create_export 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher conf.iscsi_write_cache) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py", line 544, in create_iscsi_target 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher LOG.error("%s" % e) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/i18n/_message.py", line 167, in __str__ 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher raise UnicodeError(msg) 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. 2014-12-17 20:10:46.426 1375 TRACE oslo.messaging.rpc.dispatcher 2014-12-17 20:10:46.431 1375 ERROR oslo.messaging._drivers.common [req-f1c65d49-ab3c-41f4-9764-3e6321a7c6cd 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Returning exception Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. to caller 2014-12-17 20:10:46.432 1375 ERROR oslo.messaging._drivers.common [req-f1c65d49-ab3c-41f4-9764-3e6321a7c6cd 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(args, *kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 875, in initialize_connection\n volume)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(args, *kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 548, in create_export\n return self._create_export(context, volume)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 560, in _create_export\n self.configuration)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 61, in create_export\n conf.iscsi_write_cache)\n', ' File "/usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py", line 544, in create_iscsi_target\n LOG.error("%s" % e)\n', ' File "/usr/lib/python2.7/site-packages/oslo/i18n/_message.py", line 167, in __str__\n raise UnicodeError(msg)\n', 'UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead.\n'] 2014-12-17 20:10:47.686 1375 INFO cinder.brick.iscsi.iscsi [req-752aa428-8940-4463-b3ce-c2fc0ce3a324 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Creating iscsi_target for volume: volume-eacfef77-1a5f-49e0-a13f-786d0d58c214 2014-12-17 20:11:36.518 1375 INFO cinder.volume.manager [-] Updating volume status 2014-12-17 20:11:36.707 1375 INFO cinder.volume.manager [-] Updating volume replication status. 2014-12-17 20:12:12.481 1375 INFO cinder.brick.iscsi.iscsi [req-fc4a9ec2-1d6d-46c4-a263-1bdb49db96da 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Removing iscsi_target: b27829cd-8b88-42f0-b03c-c04734e634a2 2014-12-17 20:12:13.289 1375 INFO cinder.volume.manager [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] volume b27829cd-8b88-42f0-b03c-c04734e634a2: deleting 2014-12-17 20:12:13.305 1375 INFO cinder.brick.iscsi.iscsi [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Removing iscsi_target: b27829cd-8b88-42f0-b03c-c04734e634a2 2014-12-17 20:12:13.957 1375 INFO cinder.volume.utils [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Performing secure delete on volume: /dev/mapper/cinder--volumes-volume--b27829cd--8b88--42f0--b03c--c04734e634a2 2014-12-17 20:12:19.193 1375 INFO cinder.volume.utils [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Volume copy 1024.00 MB at 207.27 MB/s 2014-12-17 20:12:20.543 1375 INFO cinder.volume.manager [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] volume b27829cd-8b88-42f0-b03c-c04734e634a2: deleted successfully 2014-12-17 20:12:20.651 1375 INFO cinder.volume.manager [req-fb10ea54-151e-4079-87f8-772617a77000 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Updating volume status 2014-12-17 20:12:36.521 1375 INFO cinder.volume.manager [-] Updating volume status 2014-12-17 20:12:36.703 1375 INFO cinder.volume.manager [-] Updating volume replication status. 2014-12-17 20:12:54.831 1375 INFO cinder.brick.iscsi.iscsi [req-41d1e524-ae3a-4e3e-9978-a5402d8089ce 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Removing iscsi_target: eacfef77-1a5f-49e0-a13f-786d0d58c214 2014-12-17 20:12:57.619 1375 INFO cinder.brick.iscsi.iscsi [req-752374bd-75d9-4851-8a07-1ccbc28cfd2e 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Creating iscsi_target for volume: volume-eacfef77-1a5f-49e0-a13f-786d0d58c214 *2014-12-17 20:12:58.351 1375 ERROR cinder.brick.iscsi.iscsi [req-752374bd-75d9-4851-8a07-1ccbc28cfd2e 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Failed to create iscsi target for volume id:volume-eacfef77-1a5f-49e0-a13f-786d0d58c214. 2014-12-17 20:12:58.354 1375 ERROR oslo.messaging.rpc.dispatcher [req-752374bd-75d9-4851-8a07-1ccbc28cfd2e 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Exception during message handling: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, *new_args) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher return f(args, *kwargs) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 875, in initialize_connection 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher volume) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher return f(args, *kwargs) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 548, in create_export 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher return self._create_export(context, volume) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 560, in _create_export 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher self.configuration) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 61, in create_export 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher conf.iscsi_write_cache) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py", line 544, in create_iscsi_target 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher LOG.error("%s" % e) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/i18n/_message.py", line 167, in __str__ 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher raise UnicodeError(msg) 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. 2014-12-17 20:12:58.354 1375 TRACE oslo.messaging.rpc.dispatcher 2014-12-17 20:12:58.359 1375 ERROR oslo.messaging._drivers.common [req-752374bd-75d9-4851-8a07-1ccbc28cfd2e 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Returning exception Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead. to caller 2014-12-17 20:12:58.360 1375 ERROR oslo.messaging._drivers.common [req-752374bd-75d9-4851-8a07-1ccbc28cfd2e 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(args, *kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 875, in initialize_connection\n volume)\n', ' File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper\n return f(args, *kwargs)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 548, in create_export\n return self._create_export(context, volume)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 560, in _create_export\n self.configuration)\n', ' File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 61, in create_export\n conf.iscsi_write_cache)\n', ' File "/usr/lib/python2.7/site-packages/cinder/brick/iscsi/iscsi.py", line 544, in create_iscsi_target\n LOG.error("%s" % e)\n', ' File "/usr/lib/python2.7/site-packages/oslo/i18n/_message.py", line 167, in __str__\n raise UnicodeError(msg)\n', 'UnicodeError: Message objects do not support str() because they may contain non-ascii characters. Please use unicode() or translate() instead.\n'] 2014-12-17 20:12:59.605 1375 INFO cinder.brick.iscsi.iscsi [req-7b3715f7-8c53-4d62-8430-4c63bb5b82bf 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Creating iscsi_target for volume: volume-eacfef77-1a5f-49e0-a13f-786d0d58c214 2014-12-17 20:13:36.521 1375 INFO cinder.volume.manager [-] Updating volume status 2014-12-17 20:13:36.714 1375 INFO cinder.volume.manager [-] Updating volume replication status. 2014-12-17 20:14:36.525 1375 INFO cinder.volume.manager [-] Updating volume status 2014-12-17 20:14:36.717 1375 INFO cinder.volume.manager [-] Updating volume replication status. 2014-12-17 20:15:06.468 1375 INFO cinder.brick.iscsi.iscsi [req-6d81513a-6385-46df-8c2a-921513754c5c 29da60f9fbd34f2582ace23177bd4424 ef9857cd8c8648e7bfd13bb555e25d6c - - -] Removing iscsi_target: eacfef77-1a5f-49e0-a13f-786d0d58c214 2014-12-17 20:15:36.528 1375 INFO cinder.volume.manager [-] Updating volume status 2014-12-17 20:15:36.713 1375 INFO cinder.volume.manager [-] Updating volume replication status.