Cinder volume mounted on rhel7 becomes read only

asked 2019-03-14 01:31:44 -0500

Anilkumar P S gravatar image

updated 2019-03-14 02:23:37 -0500

Hi All,

I have an issue w.r.t cinder volume. Volume is mounted on rhel7. It becomes read-only abruptly. We solve it by rebooting the VM. after some hours/days, it again becomes read-only.

Is there any permanent fix for the issue? What could be the possible root cause of the issue?

Logs: Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.105237] blk_update_request: I/O error, dev vdb, sector 151259488 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.109150] EXT4-fs warning (device vdb): ext4_end_bio:332: I/O error -5 writing to inode 4719235 (offset 0 size 12288 starting block 18907439) Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.109158] Buffer I/O error on device vdb, logical block 18907436 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.112672] Buffer I/O error on device vdb, logical block 18907437 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.115602] Buffer I/O error on device vdb, logical block 18907438 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.117459] blk_update_request: I/O error, dev vdb, sector 151257552 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.121200] EXT4-fs warning (device vdb): ext4_end_bio:332: I/O error -5 writing to inode 4719231 (offset 0 size 4096 starting block 18907195) Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.121206] Buffer I/O error on device vdb, logical block 18907194 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.195991] blk_update_request: I/O error, dev vdb, sector 151259640 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.199765] EXT4-fs warning (device vdb): ext4_end_bio:332: I/O error -5 writing to inode 4719233 (offset 0 size 4096 starting block 18907456) Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.199773] Buffer I/O error on device vdb, logical block 18907455 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.203598] blk_update_request: I/O error, dev vdb, sector 151257544 Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.207227] EXT4-fs warning (device vdb): ext4_end_bio:332: I/O error -5 writing to inode 4719238 (offset 0 size 4096 starting block 18907194) Mar 8 01:00:40 htcp-zhblaf-db1 kernel: [4462013.207234] Buffer I/O error on device vdb, logical block 18907193 Mar 8 01:00:45 htcp-zhblaf-db1 kernel: [4462017.547342] blk_update_request: I/O error, dev vdb, sector 319191768 Mar 8 01:00:45 htcp-zhblaf-db1 kernel: [4462017.551135] Aborting journal on device vdb-8. Mar 8 01:00:45 htcp-zhblaf-db1 kernel: [4462017.551881] EXT4-fs error (device vdb) in ext4_truncate:3629: Readonly filesystem Mar 8 01:00:46 htcp-zhblaf-db1 kernel: [4462018.691501] blk_update_request: I/O error, dev vdb, sector 222298120 Mar 8 01:00:46 htcp-zhblaf-db1 kernel: [4462018.695117] Buffer I/O error on device vdb, logical block 27787265 Mar 8 01:00:46 htcp-zhblaf-db1 kernel: [4462018.698390] lost page write due to I/O error on vdb Mar 8 01:00:46 htcp-zhblaf-db1 kernel: [4462018.732906] blk_update_request: I/O error, dev vdb, sector 112 Mar 8 01:00:46 htcp-zhblaf-db1 kernel: [4462018.736229] Buffer I/O error on device vdb, logical block 14 Mar 8 01:00 ... (more)

edit retag flag offensive close merge delete

Comments

It would be helpful if you posted log messages in a readable format. However, I see plenty of IO errors. I suppose the filesystem is set read-only to avoid corruptions.

Which Cinder backend are you using? Have you checked Cinder or Nova-Compute logs?

Bernd Bausch gravatar imageBernd Bausch ( 2019-03-14 01:53:08 -0500 )edit

Hi,

Thanks. do you know who has remounted filesystem as read-only? Is it a know issue? what could be the possible root cause of this issue?

Regards, Anil

Anilkumar P S gravatar imageAnilkumar P S ( 2019-03-14 02:37:55 -0500 )edit

I don’t know exactly what part of the kernel remounts the filesystem as read-only. I assume this is done to prevent further damage after all the IO errors.

IO errors could be caused by the connection of that volume to the VM, a faulty disk, a faulty application, a full host filesystem, etc.....

Bernd Bausch gravatar imageBernd Bausch ( 2019-03-14 03:43:22 -0500 )edit