Hi, I'm running Havana multinode. Instances and images are located on SAN attached shared disk (GPFS). Glance images need to be copied by "cp" instead of "curl" to nova's "_base" directory. Here is my configs:

/etc/glance/glance-api.conf

filesystem_store_datadir = /gpfs/images/
show_multiple_locations = True
filesystem_store_metadata_file = /etc/glance/gpfs.json

/etc/glance/gpfs.json

{
    "id": "b2b3229e-f22f-4af1-a809-fcf72afe8577",
    "mountpoint": "/gpfs"
}

/etc/nova/nova.conf

allowed_direct_url_schemes=file
filesystems=gpfs
[image_file_url:gpfs]
id=b2b3229e-f22f-4af1-a809-fcf72afe8577
mountpoint=/gpfs

Nova log on compute node

2013-12-25 17:29:15.512 10058 INFO nova.virt.libvirt.driver [req-af8fc341-bc26-4217-ba47-51a63d39a934 3cb68bbdc8bf499d82dee70392fe1c62 d1944f8305224f00b7f1faf72937f448] [instance: 4c18cdd0-c852-406
7-bef7-de0b3e6db82b] Creating image
2013-12-25 17:29:16.109 10058 ERROR nova.image.glance [req-af8fc341-bc26-4217-ba47-51a63d39a934 3cb68bbdc8bf499d82dee70392fe1c62 d1944f8305224f00b7f1faf72937f448] Unexpected error while running com
mand.
Command: cp /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315 /var/lib/nova/instances/_base/bbc9f62419d817181cf3f8f72530133bc0a1172e.part
Exit code: 1
Stdout: ''
Stderr: "cp: cannot open `/gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315' for reading: Permission denied\n"
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Traceback (most recent call last):
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 338, in download
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     xfer_mod.download(context, o, dst_path, loc_meta)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/image/download/file.py", line 164, in download
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     lv_utils.copy_image(source_file, dst_file)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 462, in copy_image
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     execute('cp', src, dest)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 50, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     return utils.execute(*args, **kwargs)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     return processutils.execute(*cmd, **kwargs)
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance     cmd=' '.join(cmd))
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance ProcessExecutionError: Unexpected error while running command.
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Command: cp /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315 /var/lib/nova/instances/_base/bbc9f62419d817181cf3f8f72530133bc0a1172e.part
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Exit code: 1
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Stdout: ''
2013-12-25 17:29:16.109 10058 TRACE nova.image.glance Stderr: "cp: cannot open `/gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315' for reading: Permission denied\n"

File permissions on image

-rw-r-----. 1 glance glance 10718478336 Dec 23 19:21 /gpfs/images/e70e8713-b96b-4e6e-85a6-eda501889315

I assume that compute service was trying ... (more)