read failed with input/output error on VolGroup

asked 2015-03-19 01:36:57 -0600

Brian.Cho gravatar image

updated 2015-03-20 00:32:15 -0600

I've installed and configured the icehouse with nova only. One controller node with glance, nova-management, keystone, horizon, rabbitmq, cinder-management and five compute nodes with cinder-volume, nova-hypervisor, kvm, nova-networking.

Also, each compute node has each cinder-volume under local hard drive with name of cinder-volumes. One of the compute node doesn't use internal hard disk to use cinder block storage. It has external storage attached by FC cable. (Hitachi) .

Under this circumstance, there was no problem at first even though i encountered several bugs and issues. But, I was going to pull through all of those issues with google. :)

The problem happened after a few days later when I recognized what openstack really is.

  1. so wired errors shows as I/O error as bellows.
  2. The name of 'VolGroup' has been created when i make the guest under compute node. It was happened only specific compute node which using internal hard disk for cinder volume. Let's compare this first.

internal hard disk

> root@compute3:~# lvscan  
> /dev/VolGroup/lv_root: read failed after 0 of 4096 at 7205748736  Input/output error  
> /dev/VolGroup/lv_root: read failed after 0 of 4096 at 7205806080: Input/output error  
> /dev/VolGroup/lv_root: read failed after 0 of 4096 at 0: Input/output error   /dev/VolGroup/lv_root: read failed after 0 of 4096 at 4096: Input/output error  
> /dev/VolGroup/lv_swap: read failed after 0 of 4096 at 855572480: Input/output error  
> /dev/VolGroup/lv_swap: read failed after 0 of 4096 at 855629824:
> Input/output error   /dev/VolGroup/lv_swap: read failed after 0 of 4096 at 0: Input/output error   /dev/VolGroup/lv_swap: read failed after 0 of 4096 at 4096: Input/output error  
> /dev/VolGroup00/LogVol00: read failed after 0 of 4096 at 6341722112: Input/output error  
> /dev/VolGroup00/LogVol00: read failed after 0 of 4096 at 6341779456: Input/output error  
> /dev/VolGroup00/LogVol00: read failed after 0 of 4096 at 0: Input/output error   
> /dev/VolGroup00/LogVol00: read failed after 0 of 4096 at 4096: Input/output error  
> /dev/VolGroup00/LogVol01: read failed after 0 of 4096 at 2113863680:
> ...    
> '/dev/VolGroup00/LogVol00' [6.88 GiB] inherit   inactive         
> '/dev/VolGroup00/LogVol01' [1.00 GiB] inherit
> ...

external storage

==> one more question from here. why /dev/rhel/root device has been created?

>root@compute2:~# lvscan
> ACTIVE            '/dev/rhel/swap' [820.00 MiB] inherit
> ACTIVE            '/dev/rhel/root' [6.71 GiB] inherit
> ACTIVE   Original '/dev/cinder-volumes/volume-1a61e3a3-56a3-4ac5-b328-002f96ca94ed' [8.00 GiB] inherit
> ACTIVE            '/dev/cinder-volumes/volume-f731506a-1e98-47da-95f9-c74a79e3ad0b' [8.00 GiB] inherit
> ACTIVE            '/dev/cinder-volumes/volume-ec9c1bc6-1f81-4ca3-b652-ec454efa85b8' [8.00 GiB] inherit
> ACTIVE            '/dev/cinder-volumes/volume-da163032-bdc5-4031-a1b8-a25835df5527' [8.00 GiB] inherit
> ACTIVE            '/dev/cinder-volumes/volume-54a85d0b-6a3b-4483-a507-588aac863912' [8.00 GiB] inherit
> ACTIVE            '/dev/cinder-volumes/volume-ce4bc764-9dba-4750-ab67-d18790c27ae1' [8.00 GiB] inherit
  1. Volume has been destroyed a few days later with I/O error. After this error I got, the linux won't be able to alive anymore. I couldn't start the vm anymore.

    [405921.262630] Buffer I/O error on device vda1, logical block 0
    [405921.341309] Aborting journal on device vda1-8.
    [405921.262630] Buffer I/O error ...
(more)
edit retag flag offensive close merge delete